Welcome to Less Wrong! (5th thread, March 2013)

post by orthonormal · 2013-04-01T16:19:17.933Z · LW · GW · Legacy · 1761 comments

Contents

  A few notes about the site mechanics
  A few notes about the community
  A list of some posts that are pretty awesome
None
1746 comments
If you've recently joined the Less Wrong community, please leave a comment here and introduce yourself. We'd love to know who you are, what you're doing, what you value, how you came to identify as a rationalist or how you found us. You can skip right to that if you like; the rest of this post consists of a few things you might find helpful. More can be found at the FAQ.

(This is the fifth incarnation of the welcome thread; once a post gets over 500 comments, it stops showing them all by default, so we make a new one. Besides, a new post is a good perennial way to encourage newcomers and lurkers to introduce themselves.)

A few notes about the site mechanics

Less Wrong comments are threaded for easy following of multiple conversations. To respond to any comment, click the "Reply" link at the bottom of that comment's box. Within the comment box, links and formatting are achieved via Markdown syntax (you can click the "Help" link below the text box to bring up a primer).

You may have noticed that all the posts and comments on this site have buttons to vote them up or down, and all the users have "karma" scores which come from the sum of all their comments and posts. This immediate easy feedback mechanism helps keep arguments from turning into flamewars and helps make the best posts more visible; it's part of what makes discussions on Less Wrong look different from those anywhere else on the Internet.

However, it can feel really irritating to get downvoted, especially if one doesn't know why. It happens to all of us sometimes, and it's perfectly acceptable to ask for an explanation. (Sometimes it's the unwritten LW etiquette; we have different norms than other forums.) Take note when you're downvoted a lot on one topic, as it often means that several members of the community think you're missing an important point or making a mistake in reasoning— not just that they disagree with you! If you have any questions about karma or voting, please feel free to ask here.

Replies to your comments across the site, plus private messages from other users, will show up in your inbox. You can reach it via the little mail icon beneath your karma score on the upper right of most pages. When you have a new reply or message, it glows red. You can also click on any user's name to view all of their comments and posts.

It's definitely worth your time commenting on old posts; veteran users look through the recent comments thread quite often (there's a separate recent comments thread for the Discussion section, for whatever reason), and a conversation begun anywhere will pick up contributors that way.  There's also a succession of open comment threads for discussion of anything remotely related to rationality.

Discussions on Less Wrong tend to end differently than in most other forums; a surprising number end when one participant changes their mind, or when multiple people clarify their views enough and reach agreement. More commonly, though, people will just stop when they've better identified their deeper disagreements, or simply "tap out" of a discussion that's stopped being productive. (Seriously, you can just write "I'm tapping out of this thread.") This is absolutely OK, and it's one good way to avoid the flamewars that plague many sites.

EXTRA FEATURES:
There's actually more than meets the eye here: look near the top of the page for the "WIKI", "DISCUSSION" and "SEQUENCES" links.
LW WIKI: This is our attempt to make searching by topic feasible, as well as to store information like common abbreviations and idioms. It's a good place to look if someone's speaking Greek to you.
LW DISCUSSION: This is a forum just like the top-level one, with two key differences: in the top-level forum, posts require the author to have 20 karma in order to publish, and any upvotes or downvotes on the post are multiplied by 10. Thus there's a lot more informal dialogue in the Discussion section, including some of the more fun conversations here.
SEQUENCES: A huge corpus of material mostly written by Eliezer Yudkowsky in his days of blogging at Overcoming Bias, before Less Wrong was started. Much of the discussion here will casually depend on or refer to ideas brought up in those posts, so reading them can really help with present discussions. Besides which, they're pretty engrossing in my opinion.

A few notes about the community

If you've come to Less Wrong to  discuss a particular topic, this thread would be a great place to start the conversation. By commenting here, and checking the responses, you'll probably get a good read on what, if anything, has already been said here on that topic, what's widely understood and what you might still need to take some time explaining.

If your welcome comment starts a huge discussion, then please move to the next step and create a LW Discussion post to continue the conversation; we can fit many more welcomes onto each thread if fewer of them sprout 400+ comments. (To do this: click "Create new article" in the upper right corner next to your username, then write the article, then at the bottom take the menu "Post to" and change it from "Drafts" to "Less Wrong Discussion". Then click "Submit". When you edit a published post, clicking "Save and continue" does correctly update the post.)

If you want to write a post about a LW-relevant topic, awesome! I highly recommend you submit your first post to Less Wrong Discussion; don't worry, you can later promote it from there to the main page if it's well-received. (It's much better to get some feedback before every vote counts for 10 karma- honestly, you don't know what you don't know about the community norms here.)

If you'd like to connect with other LWers in real life, we have  meetups  in various parts of the world. Check the wiki page for places with regular meetups, or the upcoming (irregular) meetups page. There's also a Facebook group. If you have your own blog or other online presence, please feel free to link it.

If English is not your first language, don't let that make you afraid to post or comment. You can get English help on Discussion- or Main-level posts by sending a PM to one of the following users (use the "send message" link on the upper right of their user page). Either put the text of the post in the PM, or just say that you'd like English help and you'll get a response with an email address.
* Normal_Anomaly
* Randaly
* shokwave
* Barry Cotter

A note for theists: you will find the Less Wrong community to be predominantly atheist, though not completely so, and most of us are genuinely respectful of religious people who keep the usual community norms. It's worth saying that we might think religion is off-topic in some places where you think it's on-topic, so be thoughtful about where and how you start explicitly talking about it; some of us are happy to talk about religion, some of us aren't interested. Bear in mind that many of us really, truly have given full consideration to theistic claims and found them to be false, so starting with the most common arguments is pretty likely just to annoy people. Anyhow, it's absolutely OK to mention that you're religious in your welcome post and to invite a discussion there.

A list of some posts that are pretty awesome

I recommend the major sequences to everybody, but I realize how daunting they look at first. So for purposes of immediate gratification, the following posts are particularly interesting/illuminating/provocative and don't require any previous reading:

More suggestions are welcome! Or just check out the top-rated posts from the history of Less Wrong. Most posts at +50 or more are well worth your time.

Welcome to Less Wrong, and we look forward to hearing from you throughout the site!

Note from orthonormal: MBlume and other contributors wrote the original version of this welcome post, and I've edited it a fair bit. If there's anything I should add or update on this post (especially broken links), please send me a private message—I may not notice a comment on the post. Finally, once this gets past 500 comments, anyone is welcome to copy and edit this intro to start the next welcome thread.

1761 comments

Comments sorted by top scores.

comment by atomliner · 2013-04-12T08:31:40.315Z · LW(p) · GW(p)

Hello! I call myself Atomliner. I'm a 23 year old male Political Science major at Utah Valley University.

From 2009 to 2011, I was a missionary for the Mormon Church in northeastern Brazil. In the last month I was there, I was living with another missionary who I discovered to be a closet atheist. In trying to help him rediscover his faith, he had me read The God Delusion, which obliterated my own. I can't say that book was the only thing that enabled me to leave behind my irrational worldview, as I've always been very intellectually curious and resistant to authority. My mind had already been a powder keg long before Richard Dawkins arrived with the spark to light it.

Needless to say, I quickly embraced atheism and began to read everything I could about living without belief in God. I'm playing catch-up, trying to expand my mind as fast as I can to make up for the lost years I spent blinded by religious dogma. Just two years ago, for example, I believed homosexuality was an evil that threatened to destroy civilization, that humans came from another planet, and that the Lost Ten Tribes were living somewhere underground beneath the Arctic. Needless to say, my re-education process has been exhausting.

One ex-Mormon friend of mine introduced me to Harry Potter and the Methods of Rationality, which I read only a few chapters of, but I was intrigued by the concept of Bayes Theorem and followed a link here. Since then I've read From Skepticism to Technical Rationality and many of the Sequences. I'm hooked! I'm really liking what I find here. While I may not be a rationalist now, I would really like to be.

And that's my short story! I look forward to learning more from all of you and, hopefully, contributing in the future. :)

Replies from: Eliezer_Yudkowsky, MugaSofer, Kawoomba, JohnH, private_messaging
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-12T21:14:52.699Z · LW(p) · GW(p)

Welcome to LW! Don't worry about some of the replies you're getting, polls show we're overwhelmingly atheist around here.

Replies from: MugaSofer, JohnH
comment by MugaSofer · 2013-04-12T21:39:53.022Z · LW(p) · GW(p)

This^

That said, my hypothetical atheist counterpart would have made the exact same comment. I can't speak for JohnH, but I can see someone with experience of Mormons not holding those beliefs being curious regardless of affiliation. And, of course, the other two - well, three now - comments are from professed atheists. So far nobody seems willing to try and reconvert him or anything.

comment by JohnH · 2013-04-13T17:30:49.334Z · LW(p) · GW(p)

Some of that might be because of evaporative cooling. Reading the sequences is more likely to cause a theist to ignore Less Wrong then it is to change their beliefs, regardless of how rational or not a theist is. If they get past that point they soon find Less Wrong is quite welcoming towards discussions of how dumb or irrational religion is but fairly hostile to those that try and say that religion is not irrational; as in this welcome thread even points that out.

What I am wondering about is why it seems that atheists have complete caricatures of their previous theist beliefs. What atomliner mentions as his previous beliefs has absolutely no relation to what is found in Preach My Gospel, the missionary manual that he presumably had been studying for those two years, or to anything else that is found in scripture or in the teachings of the church. So are the beliefs that he gives as what he previously believed actually what he believed and if so what did he think of the complete lack of those beliefs being found in scripture and the publications of the church that he belonged to and where did he pick up these non standard beliefs? Or is something else entirely going on when he says that those were his beliefs?

This doesn't limit itself to atomliner; in my experience generally when atheists talk about their previous religion they seem to have always held (or claim they did) some extremely non-standard version of that religion. So is this a failure of the religion to communicate what the actual beliefs are, a failure of the ex-theist to discover what the beliefs of the religion really are and think critically about, in Mormon terms, "faith promoting rumors" (also known as lies and false doctrine, in Mormon terms), or are these non-standard beliefs cobbled together from "faith promoting rumors" after the atheist is already an atheist to justify atheism?

I know that atheists can deal with a lot of prejudice from believers about why they are atheists so I would think that atheists would try and justify their beliefs based on the best beliefs and arguments of a religion and not extreme outliers for both, as otherwise it plays to the prejudice. Or at least come up with something that actually are real beliefs. For any ex-Mormon there are entire websites of ready made points of doubt which are really easy to find, there should be no need to come up with such strange outlier beliefs to justify oneself, and if justifying isn't what he is doing then I am really very interested in knowing how and why he held those beliefs.

Replies from: Eliezer_Yudkowsky, Vaniver, Estarlio, atomliner, CCC, None, MugaSofer
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-13T21:19:47.250Z · LW(p) · GW(p)

IIRC the standard experimental result is that atheists who were raised religious have substantially above-average knowledge of their former religions. I am also suspicious that any recounting whatsoever of what went wrong will be greeted by, "But that's not exactly what the most sophisticated theologians say, even if it's what you remember perfectly well being taught in school!"

This obviously won't be true in my own case since Orthodox Jews who stay Orthodox will put huge amounts of cumulative effort into learning their religion's game manual over time. But by the same logic, I'm pretty sure I'm talking about a very standard element of the religion when I talk about later religious authorities being presumed to have immensely less theological knowledge than earlier authorities and hence no ability to declare earlier authorities wrong. As ever, you do not need a doctorate in invisible sky wizard to conclude that there is no invisible sky wizard, and you also don't need to know all the sophisticated excuses for why the invisible sky wizard you were told about is not exactly what the most sophisticated dupes believe they believe in (even as they go on telling children about the interventionist superparent). It'd be nice to have a standard, careful and correct explanation of why this is a valid attitude and what distinguishes it from the attitude of an adolescent who finds out everything they were told about quantum mechanics is wrong, besides the obvious distinction of net weight of experimental evidence (though really that's just enough).

LW has reportedly been key in deconverting many, many formerly religious readers. Others will of course have fled. It takes all kinds of paths.

Replies from: MugaSofer, JohnH, Baruta07, Vaniver
comment by MugaSofer · 2013-04-14T18:49:44.763Z · LW(p) · GW(p)

As ever, you do not need a doctorate in invisible sky wizard to conclude that there is no invisible sky wizard, and you also don't need to know all the sophisticated excuses for why the invisible sky wizard you were told about is not exactly what the most sophisticated dupes believe they believe in (even as they go on telling children about the interventionist superparent).

The trouble with this heuristic is it fails when you aren't right to start with. See also: creationists.

That said, you do, in fact, seem to understand the claims theologians make pretty well, so I'm not sure why you're defending this position in the first place. Arguments are soldiers?

But by the same logic, I'm pretty sure I'm talking about a very standard element of the religion when I talk about later religious authorities being presumed to have immensely less theological knowledge than earlier authorities and hence no ability to declare earlier authorities wrong.

Well, I probably know even less about your former religion than you do, but I'm guessing - and some quick google-fu seems to confirm - that while you are of course correct about what you were thought, the majority of Jews would not subscribe to this claim.

You hail from Orthodox Judaism, a sect that contains mostly those who didn't reject the more easily-disprove elements of Judaism (and indeed seems to have developed new beliefs guarding against such changes, such as concept of a "written and oral Talmud" that includes the teachings of earlier authorites.) Most Jews (very roughly 80%) belong to less extreme traditions, and thus, presumably, are less likely to discover flaws in them. Much like the OP belonging to a subset of Mormons who believe in secret polar Israelites.

I am also suspicious that any recounting whatsoever of what went wrong will be greeted by, "But that's not exactly what the most sophisticated theologians say, even if it's what you remember perfectly well being taught in school!"

Again, imagine a creationist claiming that they were taught in school that a frog turned into a monkey, dammit, and you're just trying to disguise the lies you're feeding people by telling them they didn't understand properly! If a claim is true, it doesn't matter if a false version is being taught to schoolchildren (except insofar as we should probably stop that.) That said, disproving popular misconceptions is still bringing you closer to the truth - whatever it is - and you, personally, seem to have a fair idea of what the most sophisticated theologians are claiming in any case, and address their arguments too (although naturally I don't think you always succeed, I'm not stupid enough to try and prove that here.)

Replies from: Estarlio
comment by Estarlio · 2013-04-14T21:29:46.058Z · LW(p) · GW(p)

The trouble with this heuristic is it fails when you aren't right to start with.

Disbelieving based on partial knowledge is different from disbelieving based on mistaken belief.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-15T11:23:28.792Z · LW(p) · GW(p)

I'm not sure what you mean by this.

I mistakenly believe that learning more about something will not change my probability estimate, because the absurdity heuristic tells me it's too inferentially distant to be plausible - which has the same results if you are distant from reality and the claim is true, or correct and the claim is false.

Replies from: Estarlio
comment by Estarlio · 2013-04-15T20:19:13.213Z · LW(p) · GW(p)

I'm not sure what you mean by this.

Being mistaken about something is different from not knowing everything there is to know about it.

If I'm wrong about a subject, then I don't know everything there is to know about it (assuming I'm reasoning correctly on what I know.)

But if I don't know everything there is to know about a subject, then I'm not necessarily wrong about that subject.

The former entails the latter, but the latter does not entail the former. One doesn't need a degree in biology to correct, or be corrected, about the frog thing - anymore than one needs a degree in sky wizardy to correct or be corrected about god.

Given that you can't know everything about even relatively narrow subject areas these days, (with ~7 billion humans on Earth we turn out a ridiculous amount of stuff,) what we're really dealing with here is an issue of trust: When someone says that you need to know more to make a decision, on what grounds do you decide whether or not they're just messing you around?

There's a major dis-analogy between how the Frog-based anti-evolutionist (AE) and the atheist (AT) 's questions are going to be addressed in that regard.

When the AE challenges evolution there are obvious touching stones, ideally he's told that the frog thing never happened and given a bunch of stuff he can go look up if he's interested. When the AT challenges theology he's told that he doesn't know enough, i.e. he hasn't exhausted the search space, but he's not actually pointed at anything that addresses his concern. It's more a sort of “Keep looking until you find something. Bwahahahaaa, sucker.” response.

That happens because of what evidence does and how we get it. Say, you're trying to decide whether the Earth is flat: To discover that it's vaguely spherical doesn't take a lot of evidence. I could drive to a couple of different locations and prove it to a reasonable degree of accuracy with sticks - it would not be difficult. (Or I could ask one of my friends in another city to take a measurement for me, but regardless the underlying methodology remains more or less the same.) That's an Eratosthenes level of understanding (~200BC). To discover the shape that the Earth actually is closer to an oblate spheroid, however, you need to have at least a Newton level of understanding (~1700 AD.) to predict that it being spun ought to make it bulge around the equator.

Evidence is something like, 'that which alters the conditional probability of something being observed.' But not all evidence alters the probability to the same degree. If you're off by a lot, a little bit of evidence should let you know. The more accurate you want to get the more evidence you need. Consequently, knowledge of search spaces tends to be ordered by weightiness of evidence unless the other person is playing as a hostile agent.

Even to ask the trickier questions that need that evidence requires a deep understanding that you have tuned in from a more general understanding. The odds that you'll ask a relevant question without that understanding, just by randomly mooshing concepts together, are slim.

Now the AT probably doesn't know a lot about religion. Assuming that the atheist is not a moron just randomly mooshing concepts together, her beliefs would off by a lot; she seems likely to disagree with the theist about something fairly fundamental about how evidence is meant to inform beliefs.

So, here the AT is sitting with her really weight super-massive black hole of a reason to disbelieve - and the response from the Christian is that he doesn't know everything about god. That response is missing the references that someone who actually had a reason they could point to would have. More importantly that response claims that you need deep knowledge to answer a question that was asked with shallow knowledge.

The response doesn't even look the same as the response to the frog problem. Everyone who knows even a little bit about evolution can correct the frog fella. Whereas, to my knowledge, no Christian has yet corrected a rational atheist on his or her point of disbelief. (And if they have why aren't they singing it from the rooftops - if they have as one might call it, a knock-down argument why aren't the door to door religion salesmen opening with that?)

Strictly speaking neither of them knows everything about their subjects, or likely even very much of the available knowledge. But one clearly knows more than the other and there are things that such knowledge lets him do that the other can't; point us towards proof, answer low level questions with fairly hefty answers; and is accorded an appropriately higher trust in areas that we've not yet tested ourselves.

Of course I acknowledge the possibility that a Christian, or whoever, might be able to pull off the same stunt. But since I've never seen it, and never heard of anyone who's seen it, and I'd expect to see it all over the place if there actually was an answer lurking out there.... And since I've talked two Christians out of their beliefs in the past who'd told me that I just needed to learn more about religion and know that someone who watched that debate lost their own faith as a consequence of being unable to justify their beliefs. (Admittedly I can't verify this to you so it's just a personal proof.) It seems improbable to me that they've actually got an answer.

Of course if they have such an answer all they have to do is show it to me. In the same manner as the frog-person.

(I can actually think of one reason that someone who could prove god might choose not to: If you don't know about god, under some theologies, you can't go to hell. You can't win a really nice version of heaven either but you get a reasonable existence. They had to pull that move because they didn't want to tell people that god sent their babies went to hell.

However, this latter type of person would seem mutually exclusive with the sort of person who would be interested in telling you to look more deeply into religion to begin with. I'd imagine someone who viewed your taking on more duties to not go to hell probably ought to be in the business of discouraging you joining or investigating religion.)

Anyway, yeah. I think you can subscribe to E's heuristic quite happily even in areas where you acknowledge that you're likely to be off by a long way.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-19T14:20:34.397Z · LW(p) · GW(p)

When the AE challenges evolution there are obvious touching stones, ideally he's told that the frog thing never happened and given a bunch of stuff he can go look up if he's interested. When the AT challenges theology he's told that he doesn't know enough, i.e. he hasn't exhausted the search space, but he's not actually pointed at anything that addresses his concern. It's more a sort of “Keep looking until you find something. Bwahahahaaa, sucker.” response.

I can assure you, I have personally seen atheists make arguments that are just as misinformed as the frog thingie.

For that matter, I've seen people who don't know much about evolution but are arguing for it tell creationists that a counterpoint to their claim exists somewhere, even though they don't actually know of such a "knock-down argument". And they were right.

Also, you seem to be modelling religious people as engaging in bad faith. Am I misreading you here?

The response doesn't even look the same as the response to the frog problem. Everyone who knows even a little bit about evolution can correct the frog fella.

Sure, but that was what we call an example. Creationists often make far more complex and technical-seeming arguments, which may well be beyond the expertise of the man on the street.

Whereas, to my knowledge, no Christian has yet corrected a rational atheist on his or her point of disbelief.

Maybe I parsed this wrong. Are you saying no incorrect argument has ever been made for atheism?

(And if they have why aren't they singing it from the rooftops - if they have as one might call it, a knock-down argument why aren't the door to door religion salesmen opening with that?)

Well, many do open with what they consider to be knock-down arguments, of course. But many such arguments are, y'know, long, and require considerable background knowledge.

And since I've talked two Christians out of their beliefs in the past who'd told me that I just needed to learn more about religion and know that someone who watched that debate lost their own faith as a consequence of being unable to justify their beliefs. (Admittedly I can't verify this to you so it's just a personal proof.) It seems improbable to me that they've actually got an answer.

If you have such an unanswerable argument, why aren't you "singing it from the rooftops"?

I think you can subscribe to E's heuristic quite happily even in areas where you acknowledge that you're likely to be off by a long way.

Minor point, but you realize EY wasn't the first to make this argument? And while I did invent this counterargument, I'm far from the first to do so. For example, Yvain.

Replies from: Estarlio
comment by Estarlio · 2013-04-19T18:34:40.221Z · LW(p) · GW(p)

I can assure you, I have personally seen atheists make arguments that are just as misinformed as the frog thingie.

For that matter, I've seen people who don't know much about evolution but are arguing for it tell creationists that a counterpoint to their claim exists somewhere, even though they don't actually know of such a "knock-down argument". And they were right.

Well, that's why I said ideally. Lots of people believe evolution as a matter of faith rather than reason. I'd tend to say it's a far more easily justified faith - after all you can find the answers to the questions you're talking about very easily, or at least find the general direction they're in, and the more rational people seem almost universally to believe in it, and it networks into webs of trust that seem to allow you to actually do things with your beliefs, but it's true that many people engage with it only superficially. You'd be foolish to believe in evolution just because Joe Blogs heard that we evolved on TV. Joe Blogs isn't necessarily doing any more thinking, if that's all he'll give you to go on, than if he'd heard from his pastor that god did it all.

Joe Blogs may be able to give you good reasons for believing in something without giving you an answer on your exact point - but more generally you shouldn't believe it if all he's got in his favour is that he does and he's got unjustified faith that there must be an answer somewhere.

A heuristic tends towards truth, it's the way to bet. There are situations where you follow the heuristic and what you get is the wrong answer, but the best you can do with the information at hand.

Also, you seem to be modelling religious people as engaging in bad faith. Am I misreading you here?

I consider someone who, without good basis, tells you that there's an answer and doesn't even point you in its direction, to be acting in bad faith. That's not all religious people but it seems to me at the moment to be the set we'd be talking about here.

Sure, but that was what we call an example. Creationists often make far more complex and technical-seeming arguments, which may well be beyond the expertise of the man on the street.

Maybe so, but going back to our heuristics those arguments don't hook into a verifiable web of trust.

In case I wasn't clear earlier: I do believe that when many people believe in something with good basis they're often believing in the work of a community that produces truth according to certain methods - that what's being trusted is mostly people and little bits here and there that you can verify for yourself. What grounds do you have for trusting pastors, or whoever, know much about the world - that they're good and honest producers of truth?

Maybe I parsed this wrong. Are you saying no incorrect argument has ever been made for atheism?

No, I'm saying that to my knowledge no Christian has yet corrected someone who's reasonably rational on their reason for disbelieving.

Well, many do open with what they consider to be knock-down arguments, of course. But many such arguments are, y'know, long, and require considerable background knowledge.

Knockdown arguments about large differences of belief tend to be short, because they're saying that someone's really far off, and you don't need a lot of evidence to show that someone's a great distance out. Getting someone to buy into the argument may be more difficult if they don't believe that argument is a valid method, (and a great many people don't really,) but the argument itself should be quite small.

If someone's going to technicality you to death, that's a sign that their argument is less likely to be correct if they're applying it to a large difference of belief. Scientists noticeably don't differ on the large things - they might have different interpretations of precise matters but the weight of evidence when it comes to macroscopic things is fairly overwhelming.

If you have such an unanswerable argument, why aren't you "singing it from the rooftops"?

I don't think that people who believe in god are necessarily worse off than people who don't. If you could erase belief in god from the world, I doubt it would make a great deal of difference in terms of people behaving rationally. If anything I'd say that the reasons that religion is going out of favour have more to do with a changing moral character of society and the lack of an ability to provide a coherent narrative of hope than they do with a rise of more rationally based ideologies.

Consequently, it's not an efficient use of my time. While you can say 'low probability prior, no supporting evidence, no predictive power,' in five seconds, that's going to make people who don't have a lot of intellectual courage recoil from what you're suggesting - if they understand it at a gut level at all - and in any case teaching the tools to understand what that means can take hours. And teaching someone to bring their emotions in line with justified beliefs can take months or years on top of that. Especially if you're going to have to sit down with them and walk them through all the steps to come to a belief that they don't really want very much in the first place.

Okay, sure, 'that which can be destroyed by the truth should be' - but at what cost, in what order? Don't you have better things to do with your time than pick on Christians whose lives may even be made worse by your doing so if they don't subsequently become more rational and develop well actualised theories of happiness and so on? Can you really provide a better life than a belief in god does for them? Even if you assume that making someone disbelieve god is a low-effort task, it wouldn't be as simple as just having someone disbelieve if you were to do it to promote their interests.

If there are a more efficient way of doing it then I might be up for that, but I'm just more generally interested in raising the sanity waterline that I am with swating individual beliefs here and there.

Minor point, but you realize EY wasn't the first to make this argument? And while I did invent this counterargument, I'm far from the first to do so. For example, Yvain.

I do yes, I was made to read Dawkin's awful book a few years back in school. =p

Replies from: MugaSofer
comment by MugaSofer · 2013-04-19T19:56:05.286Z · LW(p) · GW(p)

Well, that's why I said ideally. Lots of people believe evolution as a matter of faith rather than reason.

Sorry, I was saying I agreed with them. You don't have to know every argument for a position to hold it, you just have to be right.

Mind you, I generally do learn the arguments, but I'm weird like that.

I consider someone who, without good basis, tells you that there's an answer and doesn't even point you in its direction, to be acting in bad faith. That's not all religious people but it seems to me at the moment to be the set we'd be talking about here.

I'm talking more about the set of everybody who tells you to read the literature. Sure, it's a perfectly good heuristic as long as you only use it when you're dealing with that particular subset.

What grounds do you have for trusting pastors, or whoever, know much about the world - that they're good and honest producers of truth?

Well, I was thinking more theologians, but to be fair they're as bad as philosophers. Still, they've spent millennia talking about this stuff.

No, I'm saying that to my knowledge no Christian has yet corrected someone who's reasonably rational on their reason for disbelieving.

Sorry, but I'm going to have to call No True Scotsman on this. How many theists who were rational in their reasons for believing have been corrected by atheists? How many creationists who were rational in their reasons for disbelieving in evolution have been corrected by evolutionists?

I don't think that people who believe in god are necessarily worse off than people who don't. If you could erase belief in god from the world, I doubt it would make a great deal of difference in terms of people behaving rationally.

Point.

Um ... as a rationalist and the kind of idiot who exposes themself to basilisks, could you tell me this argument? Maybe rot13 it if you're not interested in evangelizing.

I do yes, I was made to read Dawkin's awful book a few years back in school. =p

Man, I'd forgotten that was the first place I came across that. Ah, nosalgia ... terrible book, though.

Replies from: Estarlio, Estarlio
comment by Estarlio · 2013-04-28T21:24:48.549Z · LW(p) · GW(p)

Comment too long - continued from last:

Point.

Um ... as a rationalist and the kind of idiot who exposes themself to basilisks, could you tell me this argument? Maybe rot13 it if you're not interested in evangelizing.

V fhccbfr gung'f bxnl.

Gur svefg guvat abgr vf gung vs lbh ybbx ng ubj lbh trg rivqrapr, jung vg ernyyl qbrf, gura V'ir nyernql tvira bar: Ybj cevbe, (r.t. uvtu pbzcyrkvgl,) ab fhccbegvat rivqrapr. Crefbanyyl gung'f irel pbaivapvat. V erzrzore jura V jnf lbhatre, naq zl cneragf jrer fgvyy va gurve 'Tbbq puvyqera tb gb Puhepu' cunfr, zl pbhfva, jub jnf xvaqn fjrrg ba zr, fnvq gb zr 'Jul qba'g lbh jnag gb tb gb Puhepu? Qba'g lbh jnag gb tb gb urnira?' naq V nfxrq gurz 'Qba'g lbh jnag gb tb gb Aneavn? Fnzr guvat.' N ovg cvguvre creuncf ohg lbh trg gur cbvag, gur vqrn bs oryvrivat vg jvgubhg fbzrbar cbalvat hc rivqrapr unf nyjnlf orra bqq gb zr - creuncf whfg orpnhfr V jnf fb hfrq gb nqhygf ylvat ol gur gvzr V jnf byq rabhtu gb haqrefgnaq gur vqrn bs tbq ng nyy.

Ohg gur cbvag vf, bs pbhefr, jung pbafgvghgrf rivqrapr? Vg zvtug frrz yvxr gurer'f jvttyr ebbz gurer, ng yrnfg vs lbh ernyyl jnag gb or pbaivaprq bs n tbq. Bar nafjre vf gung rivqrapr qbrf fbzrguvat gb gur cebonovyvgl bs na bofreingvba - vs lbh bhgchg gur fnzr cerqvpgrq bofreingvbaf ertneqyrff bs gur rivqrapr, gura vg'f whfg n phevbfvgl fgbccre engure guna rivqrapr.

Fb, ornevat gung va zvaq: Gurer ner znal jnlf bs cuenfvat gur nethzrag sbe tbq jura lbh'er gelvat gb svyy va gung rivqrapr - frafvgvivgl gb vavgvny pbaqvgvbaf vf creuncf gur zbfg erfcrpgnoyr bar gb zl zvaq - ohg abar bs gurz frrz gb zrna n guvat jvgubhg gur sbyybj nethzrag, be nethzragf gung ner erqhpvoyr gb vg, ubyqvat:

'Gurer vf n tbq orpnhfr rirelguvat gung rkvfgf unf n pnhfr & yvxr rssrpgf ner nyvxr va gurve pnhfrf.'

Vs lbh qba'g ohl vagb gung gura, juvyr lbh'ir fgvyy tbg inevbhf jnlf gb qrsvar tbq, lbh'ir tbg ab ernfba gb. (Naq vg'f abg vzzrqvngryl pyrne ubj gubfr bgure jnlf trgf lbh nalguvat erfrzoyvat rivqrapr gung lbh pna gura tb ba gb hfr.) Rira jvgu ernfba/checbfr onfrq gurbybtvrf, yvxr Yrvoavm, gur haqreylvat nffhzcgvba vf gb nffhzr gung guvatf ner gur fnzr - 'Jung vf gehr bs [ernfbaf sbe gur rkvfgrapr bs] obbxf vf nyfb gehr bs gur qvssrerag fgngrf bs gur jbeyq, sbe gur fgngr juvpu sbyybjf vf....' Gurer ur'f nffhzvat gung obbxf unir n ernfba naq gung gur jbeyq orunirf va gur fnzr jnl, uvf npghny nethzrag tbrf ba gb nffhzr n obbx jvgu ab nhgube naq rffragvnyyl eryvrf ba gur vaghvgvba gung jr unir gung guvf jbhyq or evqvphybhf, juvpu gb zl zvaq znxrf uvf nethzrag erqhpvoyr gb gur jngpuznxre nethzrag.

Nalubb.

Lbh pna trg nebhaq gur jngpuznxre guvatl yvxr guvf:

1) Rirelguvat gung rkvfgf unf n pnhfr.

Guvf bar'f abg jbegu nethvat bire. N cevzr zbire qbrfa'g, bs vgfrys, vzcyl n fragvrag tbq va gur frafr pbzzbayl zrnag. V qba'g xabj jurgure gurer jnf be jnfa'g n cevzr zbire, V fhfcrpg jr qba'g unir gur pbaprcghny ibpnohynel gb npghnyyl gnyx nobhg perngvba rk-avuvyb va n zrnavatshy jnl.

2) Yvxr rssrpgf ner nyvxr va gurve pnhfrf.

Guvf vf gur vzcbegnag bar.

Gur nffhzcgvba vf gung lbh'ir tbg n qrfvtare va gur fnzr jnl jr qrfvta negrsnpgf - uvtu pbzcyrkvgl cerffhcbfr bar, cerfhznoyl. Ubjrire, gung qbrfa'g ernyyl yvar hc jvgu ubj vairagvba jbexf:

Vs lbh jrer whfg erylvat ba trargvp tvsgf - VD be jung unir lbh - gura lbh'q trg n erthyne qvfgevohgvba jura lbh tencurq vairagvba ntnvafg VD. Ohg lbh qba'g. Lbh qba'g trg n Qnivapv jvgubhg n Syberapr. Be ng gur irel yrnfg jvgubhg gur vagryyrpghny raivebazrag bs n Syberapr. Gur vqrn gung crbcyr whfg fvg gurer naq pbzr hc jvgu vqrnf bhg bs guva nve vf abafrafr. Gur vqrn gung lbh hfr gb perngr fbzrguvat pbzr sebz lbhe rkcrevraprf va gur jbeyq vagrenpgvat jvgu gur fgehpgher bs lbhe oenva. Vs lbh ybpx fbzrbar va frafbel qrcevingvba sbe nyy gurve yvsr, gura lbh'er abg tbvat gb trg ahpyrne culfvpf bhg bs gurz ng gur bgure raq. Tneontr va tneontr bhg.

Vs yvxr rssrpgf ernyyl ner nyvxr va gurve pnhfrf, gura lbh qba'g trg n tbq jvgubhg n jbeyq. Gur vasbezngvba sbe perngvba qbrfa'g whfg zntvpnyyl nccrne hcba cbfvgvat n perngbe. Naq vs vasbezngvba vf va jbeyqf, engure guna perngbef, nf frrzf gb or gur pnfr vs lbh'er fnlvat yvxr rssrpgf yvxr pnhfrf, gura jul cbfvg n tbq ng nyy? Gur nffhzcgvba qbrfa'g qb nal jbex - abguvat zber unir orra rkcynvarq nobhg jurer gur vasbezngvba naq fgehpgher bs gur jbeyq pnzr sebz nsgre lbh'ir znqr gur nffhzcgvba guna jnf znqr orsber.

Vg'f n snveyl cbchyne zbir va gurbybtl gb pynvz gung lbh pna'g xabj gur zvaq bs tbq. Ohg rira pnyyvat vg n zvaq znxrf n ybg bs nffhzcgvbaf - naq jura lbh fgneg erzbivat gubfr nffhzcgvbaf naq fnlvat fghss gb trg bhg bs gur nobir nethzrag yvxr 'jryy, gur vqrn jnf nyjnlf gurer, va Tbq' jung ner lbh ernyyl qbvat gung'f qvssrerag gb cbfvgvat na haguvaxvat cevzr zbire? Ubj qbrf vasbezngvba va n fgngvp fgehpgher pbafgvghgr n zvaq ng nyy?

Jurer qvq gur vasbezngvba gb trg gur jbeyq pbzr sebz? V qba'g xabj, ohg hayrff lbh pna fnl ubj tbq znqr gur jbeyq - jurer ur tbg uvf vqrnf sebz - gur cerzvfr vf whfg... gur jbeyq jbhyq ybbx gur fnzr gb lbh jurgure tbq jnf gurer be abg, fb jung lbh'er gnyxvat nobhg qbrfa'g pbafgvghgr rivqrapr bs gurve rkvfgrapr. Lbh unir gb xabj gur angher bs tbq, rira vs whfg va trareny grezf, gb qvfgvathvfu vg sebz n cevzr zbire. Fhccbfvat na ntrag va gur svefg cynpr jnf zrnag gb or jung tbg lbh bhg bs gung ceboyrz naq jura vg qbrfa'g....

Gung gb zl zvaq vf n snveyl nofbyhgr nethzrag ntnvafg tbq. Gura lbh'ir whfg tbg uvf cevbe cebonovyvgl naq jungrire culfvpny cebbsf gung fcrpvsvp eryvtvbaf cerffhcbfr, gung lbh'q irevsl ba gurve bja zrevgf; v.r. ceviryvqtrq vasbezngvba gung pbhyq bayl unir pbzr sebz zrrgvat fbzrguvat tbqyl-cbjreshy, (abar bs juvpu frrzf gb unir ghearq hc lrg.)

V qba'g xabj, znlor lbh qba'g svaq gur nethzrag pbaivapvat - gur uvg engr va gung ertneq vfa'g cnegvphyneyl uvtu. Ohg V'ir abg sbhaq n aba-snvgu-onfrq nethzrag gung guvf qbrfa'g znc bagb va fbzr sbez be nabgure lrg.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-29T22:04:42.408Z · LW(p) · GW(p)

Thank you for sharing. It was, I must say, probably the best-posed argument for atheism I've ever read, and I could probably go on for days about why it doesn't move me. So I won't.

Replies from: shminux
comment by shminux · 2013-04-29T22:50:34.634Z · LW(p) · GW(p)

Chicken!

Replies from: MugaSofer
comment by MugaSofer · 2013-05-01T18:22:53.098Z · LW(p) · GW(p)

Estarlio has specifically stated that they consider arguing over this a waste of their time. To be honest, so do I.

comment by Estarlio · 2013-04-28T21:24:35.004Z · LW(p) · GW(p)

Sorry, it's taken so long to reply. I'm easily distracted by shiny objects and the prospect of work.

Let's see:

Sorry, I was saying I agreed with them. You don't have to know every argument for a position to hold it, you just have to be right.

It seems to me at the moment that you don't know if you're right. So while you don't have to know every argument for a position to hold it, if you're interested in producing truth, it's desirable to have evidence on your side - either via the beliefs of others who have a wider array of knowledge on the subject than yourself and are good at producing truth or via knowing the arguments yourself.

Mind you, I generally do learn the arguments, but I'm weird like that.

I never have the time to learn all the arguments. Though I tend to know a reasonable number by comparison to most people I meet I suppose - not that that's saying much.

I'm talking more about the set of everybody who tells you to read the literature. Sure, it's a perfectly good heuristic as long as you only use it when you're dealing with that particular subset.

Ah, more generally then that depends on who's telling you to do it and what literature they're telling you to read. If someone's asking you to put in a fairly hefty investment of time then it seems to me that requires a fairly hefty investment of trust, sort of like Let's see some cards before we start handing over money. You don't have to see the entirety of their proof up front but if they can't provide at least a short version and haven't given you any other reason to respect their ability to find truth....

Like if gwern or someone told me that there was a good proof of god in something - I've read gwern's website and respect their reasoning - that would make me inclined to do it. If I saw priests and the like regularly making coherent arguments and they had that visible evidence in their ability to find truth, then they'd get a similar allowance. But it's like they don't want to show their cards at the moment - or aren't holding any - and whenever I've given them the allowance anyway it's turned out to be a bit of a waste. So that trust's not there for them anymore.

Well, I was thinking more theologians, but to be fair they're as bad as philosophers. Still, they've spent millennia talking about this stuff.

That's true. I just wonder - it's not well ordered or homogenous.

If everyone was writting about trivial truths then you'd expect it to mostly agree with itself - lots of people saying more or less the same stuff. If it was deep knowledge then you'd expect the deep knowledge to be on the top of the heap. Insights relevant to a widely felt need impose an ordering effect on the search space. Which is to say, lots of people know about them because they're so useful.

It's entirely possible they've just spent millennia talking about not very much at all. I mean you read Malebranche, for instance, and he was considered at the time to be doing very good work. But when you read it, it's almost infantile in its misunderstandings. If that's what passed muster it does't imply good things about what they were doing with the rest of their two thousand years or so.

I'm not sure whether that's particularly clear, reading it back. When people are talking sense then the people from previous eras don't appear to pass muster to people from modern eras. They might appear smart, but they're demonstrably wrong. If Malebranche is transparently wrong to me, and I'm not especially familiar with Christian works, nor am I the smartest man who ever lived - I've met one or two people in my life I consider as smart as myself.... That's not something that looks like an argument that's the product of thousands of years of meaningful work, or that could survive as something respectable in an environment where thousands of years of work had been put in.

Sorry, but I'm going to have to call No True Scotsman on this. How many theists who were rational in their reasons for believing have been corrected by atheists? How many creationists who were rational in their reasons for disbelieving in evolution have been corrected by evolutionists?

What difference does either of those make to the claim about atheistic rationalists? I'm not making a universal claim that all rationalists are atheistic, I'm making a claim about the group of people who are rationalists and are atheistic.

NTS would be if I said no rational atheist had, to my knowledge, ever been corrected on their point of disbelief by a Christian and you said sometihng like,

"Well, Elizer is a rationalist and he's become a Christian after hearing my really awesome argument."

And then I was all, "Well obviously Elizer's a great big poopy-head rather than a rationalist."

To my mind, Elizer and a reasonable distribution of other respectable rationalists becoming Christians in response to an argument (so that we know it's not just a random mental breakdown,) would be very hefty evidence in favour of there being a good argument for being Christian out there.

However, to answer your questions: I don't know on the creationist front, but on the Christian front I personally know of ... actually now I think of it longer I know of four, one of my friends in the US changed his mind too.

I do know of one person who's gone the other way too. But not someone that I'd considered particularly rational before they did so.

comment by JohnH · 2013-04-13T21:54:22.712Z · LW(p) · GW(p)

I believe the result is that atheists have an above average knowledge of world religions, similar to Jews (and Mormons) but I don't know of results that show they have an above average knowledge of their previous religion. Assuming most of them were Christians then the answer is possibly.

In this particular case I happen to know precisely what is in all of the official church material; I will admit to having no idea where his teachers may have deviated from church publications, hence me wondering where he got those beliefs.

I suppose I can't comment on what the average believer of various other sects know of their sects beliefs, only on what I know of their sects beliefs. Which leaves the question of plausibility that I know more then the average believer of say Catholicism or Evangelical Christianity or other groups not my own.

[edit] Eliezer, I am not exactly new to this site and have previously responded in detail to what you have written here. Doing so again would get the same result as last time.

comment by Baruta07 · 2013-04-14T19:50:23.212Z · LW(p) · GW(p)

IIRC the standard experimental result is that atheists who were raised religious have substantially above-average knowledge of their former religions.

As a Grade 11 student currently attending a catholic school (and having attended christian schools all my life) I would have to vouch for the accuracy of the statement; thanks to CCS I've learned a tremendous amount about Christianity although in my case there was a lot less Homosexuality is bad then is probably the norm and more focus on the positive moral aspects...

I currently attend Bishop Carroll HS and even though it is a catholic school I have no desire to change schools because of the alternate religious courses they offer and because it's generally a great school. From my experiences there are a ton of non-religious students as well as several more unusual religions represented. I personally would recommend the school for any HS students in Calgary wishing to have a non-standard HS experience.

comment by Vaniver · 2013-04-14T18:55:13.197Z · LW(p) · GW(p)

IIRC the standard experimental result is that atheists who were raised religious have substantially above-average knowledge of their former religions.

How much of this effect do you think is due to differences in intelligence?

comment by Vaniver · 2013-04-14T18:58:45.086Z · LW(p) · GW(p)

What I am wondering about is why it seems that atheists have complete caricatures of their previous theist beliefs.

Suppose there is diversity within a religion, on how much the sensible and silly beliefs are emphasized. If the likelihood of a person rejecting a religion is positively correlated with the religion recommending silly beliefs, then we should expect that the population of atheist converts should have a larger representation of people raised in homes where silly beliefs dominated than the population of theists. That is, standard evaporative cooling, except that the reasonable people who leave become atheists, and similarly reasonable people who are in a 'warm' religious setting can't relate. (I don't know if there is empirical support for this model or not.)

comment by Estarlio · 2013-04-14T22:15:37.453Z · LW(p) · GW(p)

I know that atheists can deal with a lot of prejudice from believers about why they are atheists so I would think that atheists would try and justify their beliefs based on the best beliefs and arguments of a religion and not extreme outliers for both, as otherwise it plays to the prejudice.

Really? It don't think it takes an exceptional degree of rationality to reject religion.

I suspect what you mean is that atheists /ought/ to justify their disbelief on stronger grounds than the silliest interpretation of their opponent's beliefs. Which is true, you shouldn't disbelieve that there's a god on the grounds that one branch of one religion told you the royal family were aliens or something - that's just an argument against a specific form of one religion not against god in general.

But I suspect the task would get no easier for religion if it were facing off against more rational individuals, who'd want the strongest form of the weakest premise. (In this case I suspect something like: What you're talking about is really complex/improbable, before we get down to talking about the specifics of any doctrine, where's your evidence that we should entertain a god at all?)

What I am wondering about is why it seems that atheists have complete caricatures of their previous theist beliefs.

Selection bias maybe? You're talking to the atheists who have an emotional investment in debating religion. I'd suspect that those who'd been exposed to the sillier beliefs would have greater investment, and that stronger rationalists would have a lower investment or a higher investment in other pursuits. Or maybe atheists tend to be fairly irrational. shrug

comment by atomliner · 2013-04-14T10:30:56.656Z · LW(p) · GW(p)

I was not trying to justify my leaving the Mormon Church in saying I used to believe in the extraordinary interpretations I did. I just wanted to say that my re-education process has been difficult because I used to believe in a lot of crazy things. Also, I'm not trying to make a caricature of my former beliefs, everything I have written here about what I used to believe I will confirm again as an accurate depiction of what was going on in my head.

I think it is a misstatement of yours to say that these beliefs have "absolutely no relation to... anything else that is found in scripture or in the teachings of the church". They obviously have some relation, being that I justified these beliefs using passages from The Family: A Proclamation to the World, Journal of Discourses and Doctrine & Covenants, pretty well-known LDS texts. I showed these passages in another reply to you.

Replies from: CCC
comment by CCC · 2013-04-14T10:47:16.692Z · LW(p) · GW(p)

They obviously have some relation, being that I justified these beliefs using passages from The Family: A Proclamation to the World, Journal of Discourses and Doctrine & Covenants, pretty well-known LDS texts. I showed these passages in another reply to you.

In all fairness, JohnH wrote his post before you showed him those passages. So that data was not available to him at the time of writing.

comment by CCC · 2013-04-13T21:29:20.485Z · LW(p) · GW(p)

Some of that might be because of evaporative cooling. Reading the sequences is more likely to cause a theist to ignore Less Wrong then it is to change their beliefs, regardless of how rational or not a theist is.

I agree intuitively with your second sentance (parsing 'beliefs' as 'religious beliefs'); but as I assign both options rather low probabilities, I suspect that it isn't enough to cause much in the way of evaporative cooling.

but fairly hostile to those that try and say that religion is not irrational

I haven't really seen that hostility, myself.

This doesn't limit itself to atomliner; in my experience generally when atheists talk about their previous religion they seem to have always held (or claim they did) some extremely non-standard version of that religion.

Hmmm. It seems likely that the non-standard forms have glaring flaws; close inspection finds the flaws, and a proportion of people therefore immediately assume that all religions are equally incorrect. Which is flawed reasoning in and of itself; if one religion is flawed, this does not imply that all are flawed.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-14T18:33:01.797Z · LW(p) · GW(p)

but fairly hostile to those that try and say that religion is not irrational

I haven't really seen that hostility, myself.

I think John means "hostility" more in the sense of "non-receptiveness" rather than actively attacking those who argue for theism.

This doesn't limit itself to atomliner; in my experience generally when atheists talk about their previous religion they seem to have always held (or claim they did) some extremely non-standard version of that religion.

Hmmm. It seems likely that the non-standard forms have glaring flaws; close inspection finds the flaws, and a proportion of people therefore immediately assume that all religions are equally incorrect.

Yup, this seems to fit.

Replies from: JohnH, CCC
comment by JohnH · 2013-04-14T19:04:30.604Z · LW(p) · GW(p)

Being called a moron seems hostile to me, just to use an example right here.

Replies from: CCC, MugaSofer, Kawoomba
comment by CCC · 2013-04-14T19:18:02.375Z · LW(p) · GW(p)

That was certainly hostile, yes. However, I take the fact that the post in question is at -10 karma to suggest that the hostility is frowned upon by the community in general.

comment by MugaSofer · 2013-04-15T11:10:40.725Z · LW(p) · GW(p)

Sorry, I should have specified "except for Kawoomba".

comment by Kawoomba · 2013-04-14T19:07:50.968Z · LW(p) · GW(p)

That which can be destroyed by the truth should be. Also, the spelling is unchanged, and I'd just seen a certain Tarantino movie.

Edit: Also, politeness has its virtues and is often more effective in achieving one's goals - yet Crocker's Rules are certainly more honest. Checking the definition of moron - at least as it pertains to that aspect of a person's belief system - I mean, who would seriously dispute its applicability, even before South Park immortalized Joseph Smith's teachings?

Replies from: PhilGoetz, CCC
comment by PhilGoetz · 2013-04-14T19:58:14.959Z · LW(p) · GW(p)

I dispute its applicability, because I've known very smart Mormons. Humans are not logic engines. It's rare to find even a brilliant person who doesn't have some blind spot.

Even if it were clinically applicable, you presented it as an in-group vs. out-group joke, which is an invitation for people from one tribe to mock people from another tribe. Its message was not primarily informational.

Crocker's Rules are not an invitation to be rude.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-14T20:05:42.280Z · LW(p) · GW(p)

I don't doubt there are Mormons with a higher IQ than myself, and more knowledgeable in many fields. Maybe the term "stupid person" is too broad, I meant it with Mormonism as the referent, and as being limited in scope to that. Yet it is disheartening that there are such obvious self-deceiving failures of reasoning, and courtesy afforded to dumb beliefs may prop up the Potemkin village, may help hide the elephant behind the curtain.

Reveal Oz in the broad daylight of reason, so that those very smart Mormons you know must address that blind spot.

Replies from: JohnH
comment by JohnH · 2013-04-14T20:09:31.232Z · LW(p) · GW(p)

Calling us morons doesn't reveal anything to reason or even attempt to force me to address what you may think of as a blind spot.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-14T20:23:15.848Z · LW(p) · GW(p)

It stands to reason that if you've successfully read even parts of the Sequences, or other rationality related materials, and yet believe in the Book of Mormon, there's little that will force you to address that blind ... area ..., so why not shock therapy. Or are you just too looking forward to your own planet / world? (Brigham Young, Journal of Discourses 18:259) Maybe that's just to be taken metaphorically though, for something, or something other?

Replies from: JohnH
comment by JohnH · 2013-04-14T21:13:12.693Z · LW(p) · GW(p)

Why go to the Journal of Discourses? D&C 132 clearly states that those that receive exaltation will be gods, the only question is whether that involves receiving a planet or just being part of the divine council. The Bible clearly states that we will be heirs and joint heirs with Christ. The Journal of Discourses is not something that most members look to for doctrine as it isn't scripture. I, and any member, am free to believe whatever I want to on the subject (or say we don't know) because nothing has been revealed on the subject of exaltation and theosis other then that.

Personally, I think there are some problems with the belief that everyone will have a planet due to some of the statements that Jesus makes in the New Testament but I could be wrong and I am not about to explain the subject here, though I may have attempted to do so in the past.

comment by CCC · 2013-04-15T08:36:06.494Z · LW(p) · GW(p)

Crocker's Rules are not an excuse for you to be rude to others. They are an invitation for others to ignore politeness when talking to you. They are not an invitation for others to be rude to you for the sake of rudeness, either; only where it enables some other aim, such as efficient transfer of information.

What you did, when viewed from the outside, is a clear example of rudeness for the sake of rudeness alone. I don't see how Crocker's rules are relevant.

comment by CCC · 2013-04-14T18:58:53.564Z · LW(p) · GW(p)

I think John means "hostility" more in the sense of "non-receptiveness" rather than actively attacking those who argue for theism.

Ah. To my mind, that would be 'neutrality', not 'hostility'.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-15T11:30:51.380Z · LW(p) · GW(p)

Ironically, this turned out not to be the case; he was thinking of Kawoomba, our resident ... actually, I'd assumed he only attacked me on this sort of thing.

Replies from: CCC
comment by CCC · 2013-04-15T17:32:23.629Z · LW(p) · GW(p)

Ironically, this turned out not to be the case

A common problem when one person tries to explain the words of another to a third party, yes.

Funny thing - I had a brief interaction over private messaging with Kawoomba on the subject of religion some time back, and he seemed reasonable at the time. Mildly curious, firmly atheistic, and not at all hostile.

I'm not sure if he changed, or if he's hostile to only a specific subcategory of theists?

Replies from: MugaSofer
comment by MugaSofer · 2013-04-19T14:25:23.188Z · LW(p) · GW(p)

As I said, I'd assumed it was just me; we got into a rather lengthy argument some time ago on whether human ethics generalize, and he's been latching onto anything I say that's even tangentially related ever since. I'm not sure why he's so eager to convince me, since he believes his values are incompatible with mine, but it seems it may have something to do with him pattern-matching my position with the Inquisition or something.

comment by [deleted] · 2013-04-14T20:10:15.920Z · LW(p) · GW(p)

Have you noticed any difference between first and second generation atheists, in regard to caricaturing or contempt for religion?

comment by MugaSofer · 2013-04-13T18:15:53.624Z · LW(p) · GW(p)

Reading the sequences is more likely to cause a theist to ignore Less Wrong then it is to change their beliefs, regardless of how rational or not a theist is.

Really? I would have expected most aspiring rationalists who happen to be theists to be mildly irritated by the anti-theism bits, but sufficiently interested by the majority that's about rationality. Might be the typical mind fallacy, though.

This doesn't limit itself to atomliner; in my experience generally when atheists talk about their previous religion they seem to have always held (or claim they did) some extremely non-standard version of that religion.

I would assume this is because the standard version of major religions likely became so by being unusually resistant to deconversion - including through non-ridiculousness.

EDIT: also, I think those were intended as examples of things irrational people believe, not necessarily Mormons specifically.

Replies from: TheOtherDave
comment by TheOtherDave · 2013-04-13T18:44:37.427Z · LW(p) · GW(p)

I would have expected most aspiring rationalists who happen to be theists to be mildly irritated by the anti-theism bits

Well, I don't strongly identify as a theist, so it's hard for me to have an opinion here.

That said, if I imagine myself reading a variant version of the sequences (and LW discourse more generally) which are anti-some-group-I-identify-with in the same ways.... for example, if I substitute every reference to the superiority of atheism to theism (or the inadequacy of theism more generally) with a similar reference to the superiority of, say, heterosexuality to homosexuality (or the inadequacy of homosexuality more generally), my emotional response is basically "yeah, fuck that shit."

Perhaps that's simply an indication of my inadequate rationality.

This doesn't limit itself to atomliner; in my experience generally when atheists talk about their previous religion they seem to have always held (or claim they did) some extremely non-standard version of that religion.

I would assume this is because the standard version of major religions likely became so by being unusually resistant to deconversion - including through non-ridiculousness.

That's possible. Another possibility is that when tribe members talk about their tribe, they frequently do so charitably (for example, in nonridiculous language, emphasizing the nonridiculous aspects of their tribe), while when ex-members talk about their ex-tribe, the frequently do so non-charitably.

This is similar to what happens when you compare married people's descriptions of their spouses to divorced people's descriptions of their ex-spouses... the descriptions are vastly different, even if the same person is being described.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-13T20:22:06.394Z · LW(p) · GW(p)

Well, I don't strongly identify as a theist, so it's hard for me to have an opinion here.

That said, if I imagine myself reading a variant version of the sequences (and LW discourse more generally) which are anti-some-group-I-identify-with in the same ways.... for example, if I substitute every reference to the superiority of atheism to theism (or the inadequacy of theism more generally) with a similar reference to the superiority of, say, heterosexuality to homosexuality (or the inadequacy of homosexuality more generally), my emotional response is basically "yeah, fuck that shit."

Perhaps that's simply an indication of my inadequate rationality.

I can confirm that it is indeed annoying, and worse still can act to reduce the persuasiveness of a point (for example, talking about how large groups of people/experts/insert other heuristic here fails with regard to religion.) Interestingly, it's annoying even if I agree with the criticism in question, which would suggest it's probably largely irrational, and certain rationality techniques reduce it, like the habit of ironmanning people's points by, say, replacing "religion" with racism or the education system or some clinically demonstrated bias or whatever.

That's possible. Another possibility is that when tribe members talk about their tribe, they frequently do so charitably (for example, in nonridiculous language, emphasizing the nonridiculous aspects of their tribe), while when ex-members talk about their ex-tribe, the frequently do so non-charitably.

This is similar to what happens when you compare married people's descriptions of their spouses to divorced people's descriptions of their ex-spouses... the descriptions are vastly different, even if the same person is being described.

There's probably a bit of that too, but (in my experience) most atheists believed an oddly ... variant ... version of their faith, whether it's because they misunderstood as a child or simply belonged to a borderline splinter group. Mind you, plenty of theists are the same, just unexamined.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-13T21:05:14.215Z · LW(p) · GW(p)

for example, if I substitute every reference to the superiority of atheism to theism (or the inadequacy of theism more generally) with a similar reference to the superiority of, say, heterosexuality to homosexuality (or the inadequacy of homosexuality more generally), my emotional response is basically "yeah, fuck that shit."

These examples are not at all analogous. Claims about the existence of divine agents - or the accuracy of old textbooks - are epistemological claims about the world and not up to personal preferences. What do I know and how do I know it?

Claims about preferences can by definition not be objectively right or wrong, but only be accurate or inaccurate relative to their frame of reference, to the agent they are ascribed to. Even if that agent were some divine entity. Jesus would like you to do X, but Bob wouldn't.

Or, put differently:

"There is a ball in the box" - Given the same evidence, Clippy and an FAI will come to the same conclusion. Personal theist claims mostly fall in this category ("This book was influenced by being X", "the universe was created such-and-such", "I was absolved from my sins by a god dying for me").

"I prefer a ball in the box over no ball in the box" - Given the same evidence, rational actors do not have to agree, their preferences can be different. Sexual preferences, for example.

The reason that theists are generally regarded as irrational in their theism is because there is no reason to privilege the hypothesis that any particular age old cultural text somehow accurately describes important aspects of the universe, even if you'd ascribe to some kind of first mover. Like watching William Craig debates, who goes from some vague "First Cause" argument all the way to "the Bible is right because of the ?evidence? of a supernatural resurrection". That's a long, long way to skip and gloss over. Arguing for a first mover (no restriction other than "something that started the rest") is to arguing for the Abrahamic god what predicting the decade of your time of death would be to predicting the exact femtosecond of your death.

Such motivated cognition compromises many other aspects of one's reasoning unless it's sufficiently cordoned off, just like an AI that steadfastly insisted that human beings are all made of photons, and needed to somehow warp all its other theories to accommodate that belief.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-13T21:37:35.727Z · LW(p) · GW(p)

Well, this explains the mystery of why that got downvoted by someone.

for example, if I substitute every reference to the superiority of atheism to theism (or the inadequacy of theism more generally) with a similar reference to the superiority of, say, heterosexuality to homosexuality (or the inadequacy of homosexuality more generally), my emotional response is basically "yeah, fuck that shit."

Firstly, you're replying to an old version of my comment - the section you're replying to is part of a quote which had a formatting error, which is why it forms a complete non-sequitur taken as a reply. I did not write that, I merely replied to it.

These examples are not at all analogous. Claims about the existence of divine agents - or the accuracy of old textbooks - are epistemological claims about the world and not up to personal preferences.

You know, I agree with you, homosexuality isn't a great example there. However, it's trivially easy to ironman as "homosexuality is moral" or some other example involving the rationality skills of the of the general populace.

Claims about preferences can by definition not be objectively right or wrong, but only be accurate or inaccurate relative to their frame of reference, to the agent they are ascribed to. Even if that agent were some divine entity. Jesus would like you to do X, but Bob wouldn't.

The fact that something is true only relative to a frame of reference does not mean it "can by definition not be objectively right or wrong". For example, if I believe it is correct (by my standards) to fly a plane into a building full of people, I am objectively wrong - this genuinely, verifiably doesn't satisfy my preferences. I may have been persuaded a Friendly superintelligence has concluded that it is, or that it will cause me to experience subjective bliss (OK, this one is harder to prove outright, we could be in a simulation run by some very strange people. It is, however, irrational to believe it based on the available evidence.)

"There is a ball in the box" - Given the same evidence, Clippy and an FAI will come to the same conclusion. Personal theist claims mostly fall in this category ("This book was influenced by being X", "the universe was created such-and-such", "I was absolved from my sins by a god dying for me").

"I prefer a ball in the box over no ball in the box" - Given the same evidence, rational actors do not have to agree, their preferences can be different.

Ayup.

Sexual preferences, for example.

As I said earlier, it's trivially easy to ironman that reference to mean one of the political positions regarding the sexual preference. If he had said "abortion", would you tell him that a medical procedure is a completely different thing to an empirical claim?

The reason that theists are generally regarded as irrational in their theism is because there is no reason to privilege the hypothesis that any particular age old cultural text somehow accurately describes important aspects of the universe, even if you'd ascribe to some kind of first mover.

Forgive me if I disagree with that particular empirial claim about how our community thinks.

Like watching William Craig debates, who goes from some vague "First Cause" argument all the way to "the Bible is right because of the ?evidence? of a supernatural resurrection".

"The Bible is right because of the evidence of a supernatural resurrection" is an argument in itself, not something one derives from the First Cause. However, the prior of supernatural resurrections might be raised by a particular solution to the First Cause problem, I suppose, requiring that argument to be made first.

Arguing for a first mover (no restriction other than "something that started the rest") is to arguing for the Abrahamic god what predicting the decade of your time of death would be to predicting the exact femtosecond of your death.

I guess I can follow that analogy - you require more evidence to postulate a specific First Mover than the existence of a generalized First Cause - but I have no idea how it bears on your misreading of my comment.

Such motivated cognition compromises many other aspects of one's reasoning unless it's sufficiently cordoned off, just like an AI that steadfastly insisted that human beings are all made of photons, and needed to somehow warp all its other theories to accommodate that belief.

Source? I find most rationalists encounter more irrational beliefs being protected off from rational ones than the inverse.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-14T08:22:03.766Z · LW(p) · GW(p)

"homosexuality is moral"

How is that example any different, how is it not also a matter of your individual moral preferences? Again, you can imagine a society or species of rational agents that regard homosexuality as moral, just as you can imagine one that regards it as immoral.

The fact that something is true only relative to a frame of reference does not mean it "can by definition not be objectively right or wrong".

By objectively right or wrong I meant right or wrong regardless of the frame of reference (as it's usually interpreted as far as I know). Of course you can be mistaken about your own preferences, and other agents can be mistaken when describing your preferences.

"Agent A has preference B" can be correct or incorrect / right or wrong / accurate or inaccurate, but "Preference B is moral, period, for all agents" would be a self-contradictory nonsense statement.

If he had said "abortion", would you tell him that a medical procedure is a completely different thing to an empirical claim?

Of course "I think abortion is moral" can widely differ from rational agent to rational agent. Clippy talking to AbortAI (the abortion maximizing AI) could easily agree about what constitutes an abortion, or how that procedure is usually done. Yet they wouldn't need to agree about the morality each of them ascribes to that procedure. They would need to agree on how others ("this human in 21th century America") morally judge abortion, but they could still judge it differently. It is like "I prefer a ball in the box over no ball in the box", not like "There is a ball in the box".

Forgive me if I disagree with that particular empirial claim about how our community thinks.

I forgive you, though I won't die for your sins.

"The Bible is right because of the evidence of a supernatural resurrection" is an argument in itself.

It is ... an argument ... strictly formally speaking. What else could explain some eye witness testimony of an empty grave, if not divine intervention?

However, the prior of supernatural resurrections might be raised by a particular solution to the First Cause problem.

Only when some nonsense about "that cause must be a non-physical mind" (without defining what a non-physical mind is, and reaching that conclusion by saying "either numbers or a mind could be first causes, and it can't be numbers") is dragged in, even then the effect on the prior of some particular holy text on some planet in some galaxy in some galactic cluster would be negligible.

but I have no idea how it bears on your misreading of my comment.

"I can confirm that it is indeed annoying", although I of course admit that this is branching out on a tangent - but why shouldn't we, it's a good place for branching out without having to start a new topic, or PMs.

Not everything I write needs to be controversial between us, it can be related to a comment I respond to, and you can agree or disagree, engage or disengage at your leisure.

I find most rationalists encounter more irrational beliefs being protected off from rational ones than the inverse.

What do you mean, protected off in the sense of compartmentalized / cordoned off?

Replies from: MugaSofer
comment by MugaSofer · 2013-04-14T16:43:38.143Z · LW(p) · GW(p)

How is that example any different, how is it not also a matter of your individual moral preferences? Again, you can imagine a society or species of rational agents that regard homosexuality as moral, just as you can imagine one that regards it as immoral.

We seem to be using "moral" differently. You're using it to refer to any preference, whereas I'm using it to refer to human ethical preferences specifically. I find this is more useful, for the reasons EY puts forth in the sequences.

By objectively right or wrong I meant right or wrong regardless of the frame of reference (as it's usually interpreted as far as I know). Of course you can be mistaken about your own preferences, and other agents can be mistaken when describing your preferences.

If you can be mistaken - objectively mistaken - then you are in a state known as "objectively wrong", yes?

Of course "I think abortion is moral" can widely differ from rational agent to rational agent. Clippy talking to AbortAI (the abortion maximizing AI) could easily agree about what constitutes an abortion, or how that procedure is usually done. Yet they wouldn't need to agree about the morality each of them ascribes to that procedure. They would need to agree on how others ("this human in 21th century America") morally judge abortion, but they could still judge it differently. It is like "I prefer a ball in the box over no ball in the box", not like "There is a ball in the box".

Again, I think we're arguing over terminology rather than meaning here.

I forgive you, though I won't die for your sins.

Zing!

It is ... an argument ... strictly formally speaking. What else could explain some eye witness testimony of an empty grave, if not divine intervention?

Because that's the only eyewitness testimony contained in the Bible.

Only when some nonsense about "that cause must be a non-physical mind" (without defining what a non-physical mind is, and reaching that conclusion by saying "either numbers or a mind could be first causes, and it can't be numbers") is dragged in, even then the effect on the prior of some particular holy text on some planet in some galaxy in some galactic cluster would be negligible.

Well, since neither of actually have a solution to the First Cause argument (unless you're holding out on me) that's impossible to say. However, yes, if you believed that the solution involved extra-universal superintelligence, it would raise the prior of someone claiming to be such a superintelligence and exhibiting apparently supernatural power being correct in these claims.

"I can confirm that it is indeed annoying", although I of course admit that this is branching out on a tangent - but why shouldn't we, it's a good place for branching out without having to start a new topic, or PMs.

What does the relative strength of evidence required for various "godlike" hypotheses have to do with the annoyance of seeing a group you identify with held up as an example of something undesirable?

Not everything I write needs to be controversial between us, it can be related to a comment I respond to, and you can agree or disagree, engage or disengage at your leisure.

Uh ... sure ... I don't exactly reply to most comments you make.

What do you mean, protected off in the sense of compartmentalized / cordoned off?

Yup.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-18T09:56:49.948Z · LW(p) · GW(p)

You're using it to refer to any preference, whereas I'm using it to refer to human ethical preferences specifically.

Which humans? Medieval peasants? Martyrs? Witch-torturers? Mercenaries? Chinese? US-Americans? If so, which party, which age-group?

If you can be mistaken - objectively mistaken - then you are in a state known as "objectively wrong", yes?

The term is overloaded. I was referring to ideas such as e.g. moral universalism. An alien society - or really just different human societies - will have their own ethical preferences, and while they or you can be wrong in describing those preferences, they cannot be wrong in having them, other than their preferences being incompatible with someone else's preferences. There is no universal reference frame, even if a god existed, his preferences would just amount to an argument from authority.

However, yes, if you believed that the solution involved extra-universal superintelligence, it would raise the prior of someone claiming to be such a superintelligence and exhibiting apparently supernatural power being correct in these claims.

Negligibly so, especially if it's non verifiable second hand stories passed down through the ages, and when the whole system is ostentatiously based on non-falsifiability in an empirical sense.

You realize that your fellow Christians from a few centuries back would burn you for heresy if you told them that many of the supernatural magic tricks were just meant as metaphors. Copernicus didn't doubt Jesus Christ was a god-alien-human. They may not even have considered you to be a Christian. Nevermind that, the current iteration has gotten it right, doesn't it? Your version, I mean.

Because that's the only eyewitness testimony contained in the Bible.

There are three little pigs who saw the big bad wolf blowing away their houses, that's three eyewitnesses right there.

Do Adam and Eve count as eyewitnesses for the Garden of Eden?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-18T10:31:24.676Z · LW(p) · GW(p)

The term is overloaded. I was referring to ideas such as e.g. moral universalism. An alien society - or really just different human societies - will have their own ethical preferences, and while they or you can be wrong in describing those preferences, they cannot be wrong in having them, other than their preferences being incompatible with someone else's preferences. There is no universal reference frame, even if a god existed, his preferences would just amount to an argument from authority.

OK. So moral realism is false, and moral relativism is true and that's provable in a paragraph. Hmmm. Aliens and other societies might have all sorts of values, but that does not necessarily mean they have all sorts of ethical values. "Murder is good" might not be a coherent ethical principle, any more than "2+2=5" is a coherent mathematical one. The says-so of authorities, or Authorities is not the only possible source of objectivity.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-18T10:53:14.613Z · LW(p) · GW(p)

So if you constructed an artificial agent, you would somehow be stopped from encoding certain actions and/or goals as desirable? Or that agent would just be wrong when describing his own preferences when he then tells you "killing is good"?

Certain headwear must be worn by pious women. Light switches must not be used on certain days by god-abiding men. Infidels must be killed. All of those are ethical from even some human's frame of reference. Seems pretty variable.

Replies from: PrawnOfFate, MugaSofer
comment by PrawnOfFate · 2013-04-18T10:56:58.141Z · LW(p) · GW(p)

Or that agent would just be wrong when describing his own preferences when he then tells you "killing is good"?

It would be correctly describing its preferences, and its preferences would not be ethically correct. You could construct an AI that frimly believed 2+2=5. And it would be wrong. As before, you are glibly assuming that the word "ethical" does no work, and can be dropped from the phrase "ethical value".

Certain headwear must be worn by pious women. Light switches must not be used on certain days by god-abiding men. Infidels must be killed. All of those are ethical from even some human's frame of reference.

All of those are believed ethical. It's very shallow to argue for relativism by ignoring the distinction between believed-to-be-true and true.

Replies from: Kawoomba, Estarlio
comment by Kawoomba · 2013-04-18T11:12:24.557Z · LW(p) · GW(p)

Imagine a mirror world, inhabited by our "evil" (from our perspective) twins. Now they all go around being all unethical, yet believing themselves to act ethically. They have the same model of physics, the same technological capabilities, they'd just be mistaken about being ethical.

Could it be that it turns out that we're that unethical mirror world, and our supposedly evil twins do in fact have it right? Do you think to know at least some of what's universally ethical, or could you unknowingly be the evil twin believing to be ethical?

Or could both us and our mirror world be unethical, and really only a small cluster of sentient algae somewhere in the UDFy-38135539 galaxy has by chance gotten it right, and is acting ethically?

All advanced societies will agree about 2+2!=5, because that's falsifiable. Who gets to set the axioms and rules for ethicality? Us, the mirror world, the algae, god?

Replies from: ArisKatsaris, PrawnOfFate, ArisKatsaris
comment by ArisKatsaris · 2013-04-18T13:03:27.231Z · LW(p) · GW(p)

Who gets to set the axioms and rules for ethicality?

Axioms are what we use to logically pinpoint what it is we are talking about. If our world and theirs has different axioms for "ethicality", then they simply don't have what we mean by "ethicality" -- and we don't have what they mean by the word "ethicality".

Our two worlds would then not actually disagree about ethics the concept, they instead disagree about "ethics" the word, much like 'tier' means one thing in English and another thing in german.

Replies from: Creutzer, Kawoomba, MugaSofer, PrawnOfFate
comment by Creutzer · 2013-04-19T12:35:00.869Z · LW(p) · GW(p)

Unfortunately, words of natural language have the annoying property that it's often very hard to tell if people are disagreeing about the extension or the meaning. It's also hard to tell what disagreement about the meaning of a word actually is.

Our two worlds would then not actually disagree about ethics the concept, they instead disagree about "ethics" the word, much like 'tier' means one thing in English and another thing in german.

The analogy is flawed. German and English speakers don't disagree about the word (conceived as a string of phonemes; otherwise "tier" and "Tier" are not identical), and it's not at all clear that disagreement about the meaning of words is the same thing as speaking two different languages. It's certainly phenomenologically pretty different.

I do agree that reducing it to speaking different languages is one way to dissolve disagreement about meaning. But I'm not convinced that this is the right approach. Some words are in acute danger of being dissolved with the question in that it will turn out that almost everyone has their own meaning for the word, and everybody is talking past each other. It also leaves you with a need to explain where this persistent illusion that people are disagreeing when they're in fact just talking past each other (which persists even when you explain to them that they're just speaking two different languages; they'll often say no, they're not, they're speaking the same language but the other person is using the word wrongly) comes from.

Of course, all of this is connected to the problem that nobody seems to know what kind of thing a meaning is.

comment by Kawoomba · 2013-04-18T13:38:01.222Z · LW(p) · GW(p)

So there is an objective measure for what's "right" and "wrong" regardless of the frame of reference, there is such a thing as correct, individual independent ethics, but other people may just decide not to give a hoot, using some other definition of ethics?

Well, let's define a series of ethics, from ethics1 to ethicsn. Let's call your system of ethics which contains a "correct" conclusion such as "murder is WONG", say, ethics211412312312.

Why should anyone care about ethics211412312312?

(If you don't mind, let's consolidate this into the other sub-thread we have going.)

Replies from: PrawnOfFate, nshepperd
comment by PrawnOfFate · 2013-04-18T14:22:25.376Z · LW(p) · GW(p)

but other people may just decide not to give a hoot, using some other definition of ethics

If what they have can't do what ethics is supposed to do, why call it ethics?

Replies from: Kawoomba
comment by Kawoomba · 2013-04-18T14:23:16.707Z · LW(p) · GW(p)

What is ethics supposed to do?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-18T14:38:31.547Z · LW(p) · GW(p)

Reconcile one's preferences with those of others.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-18T14:47:37.430Z · LW(p) · GW(p)

That's one specific goal that you ascribe to your ethics-subroutine, the definition entails no such ready answer.

Ethics:

"Moral principles that govern a person's or group's behavior"

"The moral correctness of specified conduct"

Moral:

"of or relating to principles of right and wrong in behavior"

What about Ferengi ethics?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-18T14:57:46.395Z · LW(p) · GW(p)

I don't know what you mean. Your dictionary definitions are typically useless for philosophical purposes.

ETA

What about Ferengi ethics?

Well...what?

Replies from: Kawoomba
comment by Kawoomba · 2013-04-18T14:59:08.104Z · LW(p) · GW(p)

You are saying "the (true, objective, actual) purpose of ethics is to reconcile one's preferences with those of others".

Where do you take that from, and what makes it right?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-18T15:03:01.559Z · LW(p) · GW(p)

I got it from thinking and reading. It might not be right. It's a philosophical claim. Feel free to counterargue.

comment by nshepperd · 2013-04-18T13:47:45.024Z · LW(p) · GW(p)

Why should anyone care about ethics211412312312?

"Should" is an ethical word. To use your (rather misleading) naming convention, it refers to a component of ethics211412312312.

Of course one should not confuse this with "would". There's no reason to expect an arbitrary mind to be compelled by ethics.

Replies from: PrawnOfFate, Kawoomba
comment by PrawnOfFate · 2013-04-18T14:23:22.619Z · LW(p) · GW(p)

"Should" is an ethical word

No. it's much wider than that. There are rational and instrumental should's.

ETA:

here's no reason to expect an arbitrary mind to be compelled by ethics.

Depends how arbitrary. Many philosophers think a rational mind could be compelled by ethical arguments...that ethical-should can be built out of rational-should.

comment by Kawoomba · 2013-04-18T14:02:25.507Z · LW(p) · GW(p)

There's no reason to expect an arbitrary mind to be compelled by ethics.

As one should not expect an arbitrary mind with its own notions of "right" or "wrong" to yield to any human's proselytizing about objectively correct ethics, "murder is bad", and trying to provide a "correct" solution for that arbitrary mind to adopt.

The ethics as defined by China, or an arbitrary mind, have as much claim to be correct as ours. There is no axiom-free metaethical framework which would provide the "should" in "you should choose ethics211412312312", that was my point. Calling some church's (or other group's) ethical doctrine objectively correct for all minds doesn't make a dint of difference, and doesn't go beyond "my ethics are right! no, mine are!"

Replies from: PrawnOfFate, ArisKatsaris
comment by PrawnOfFate · 2013-04-18T14:29:45.518Z · LW(p) · GW(p)

As one should not expect an arbitrary mind with its own notions of "right" or "wrong" to yield to any human's proselytizing about objectively correct ethics, "murder is bad", and trying to provide a "correct" solution for that arbitrary mind to adopt.

But humans can proselytise each other, despite their different notions of right and wrong. You seem to be assuming that morally-rght and -wrong are fundamentals. But if they are outcomes of reasoning and facts, then they can be changed by the presentation of better reasoning and previously unknown facts. As happens when one person morally exhorts another. I think you need to assume that your arbitrary mind has nothing in common with a human one, not even rationality.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-18T14:34:20.542Z · LW(p) · GW(p)

But if they are outcomes of reasoning and facts, then they can be changed by the presentation of better reasoning (...) I think you need to assume that your arbitrary mind has nothing in common with a human one, not even rationality

Does that mean that, in your opinion, if we constructed an AI mind that uses a rational reasoning mechanism (such as Bayes), we wouldn't need to worry since we could persuade it to act morally correct?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-18T14:41:36.219Z · LW(p) · GW(p)

I'm not sure if that is necessarily true, or even highly likely. But it is a possibility which is extensively discussed in non-LW philosophy that is standardly ignored or bypassed on LW for some reason. As per my original comment. Is moral relativism really just obviously true?

Replies from: MugaSofer
comment by MugaSofer · 2013-04-19T12:30:13.695Z · LW(p) · GW(p)

Depends on how you define "moral relativism". Kawomba thinks a particularly strong version is obviously true, but I think the LW consensus is that a weak version is.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-19T12:47:50.820Z · LW(p) · GW(p)

I don't think there is a consensus, just a belief in a consensus. EY seems unable or unwiing to clarify his posiition even when asked directly.

comment by ArisKatsaris · 2013-04-18T14:18:19.352Z · LW(p) · GW(p)

The ethics as defined by China, or an arbitrary mind, have as much claim to be correct as ours.

If someone defines ethics differently, then WHAT are the common characteristics that makes you call them both "ethics"? You surely don't mean that they just happened to use the same sound or the same letters and that they may be meaning basketball instead? So there must already exist some common elements you are thinking of that make both versions be logically categorizable as "ethics".

What are those common elements?

What would it mean for an alien to e.g. define "tetration" differently than we do? Either they define it in the same way, or they haven't defined it at all. To define it differently means that they're not describing what we mean by tetration at all.

comment by MugaSofer · 2013-04-19T12:24:09.095Z · LW(p) · GW(p)

Cannot upvote enough.

Also, pretty sure I've made this exact argument to Kawoomba before, but I didn't phrase it as well, so good luck!

comment by PrawnOfFate · 2013-04-18T14:21:11.834Z · LW(p) · GW(p)

Axioms are what we use to logically pinpoint what it is we are talking about.

Axioms have a lot to do with truth, and little to do with meaning.

Replies from: ArisKatsaris
comment by ArisKatsaris · 2013-04-18T14:35:23.497Z · LW(p) · GW(p)

Axioms have a lot to do with truth, and little to do with meaning.

Would that make the Euclidean axioms just "false" according to you, instead of meaningfully defining the concept of a Euclidean space that turned out not to be completely corresponding to reality, but is still both quite useful and certainly meaningful as a concept?

I first read the concept of axioms as means of logical pinpointing in this and it struck me as brilliant insight which may dissolve a lot of confusions.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-18T14:36:34.696Z · LW(p) · GW(p)

Corresponding to reality is physical truth, not mathematical truth.

comment by PrawnOfFate · 2013-04-18T14:19:34.620Z · LW(p) · GW(p)

Could it be that it turns out that we're that unethical mirror world, and our supposedly evil twins do in fact have it right? Do

If relativism is true, yes. If realism is true no. So?

Or could both us and our mirror world be unethical, and really only a small cluster of sentient algae somewhere in the UDFy-38135539 galaxy has by chance gotten it right, and is acting ethically?

If realism is true, they could have got it right by chance, although whoever is right is more likely to be right by approaching it systematically.

All advanced societies will agree about 2+2!=5, because that's falsifiable.

Inasmuch as it is disproveable from non-arbitrary axioms. You are assuming that maths has non-arbitrary axioms, but morality doesn't. Is that reasonable?

Who gets to set the axioms and rules for ethicality? Us, the mirror world, the algae, god?

Axioms aren't true or false because of who is "setting" them. Maths is supposed to be able to do certain things, it is supposed to allow you to prove theorems, it is supposed to be free from contradiction and so on. That considerably constrains the choice of axioms. Non-euthyphric moral realism works the same way.

comment by ArisKatsaris · 2013-04-18T12:43:52.161Z · LW(p) · GW(p)

Imagine a mirror world, inhabited by our "evil" (from our perspective) twins. Now they all go around being all unethical, yet believing themselves to act ethically. They have the same model of physics, the same technological capabilities, they'd just be mistaken about being ethical.

Okay, let's try to figure out how that would work. A world where preferences are the same (e.g. everyone wants to live as long as possible, and wants other people to live as well), but the ethics are reversed (saving lives is considered morally wrong, murdering other people at random is morally right)

Don't you see an obvious asymmetry here between their world and ours? Their so-called ethics about murder (murder=good) would end up harming their preferences, in a way that our ethics about murder (murder=bad) does not?

Replies from: Kawoomba, TheOtherDave
comment by Kawoomba · 2013-04-18T13:33:44.605Z · LW(p) · GW(p)

So is it a component of the "correct" ethical preferences that they satisfy the preferences of others? It seems this way since you use this to hold "our" ethics about murder over those of the mirror world (In actuality there'd be vast swaths of peaceful coexistence in the mirror world, e.g. in Ruanda).

But hold on, our ethical preferences aren't designed to maximize other sapients' preferences. Wouldn't it be more ethical still to not want anything for yourself, or to be happy to just stare at the sea floor, and orient those around you to look at the sea floor as well? Seems like those algae win, after all! God's chosen seaweed!

What about when a quadrillion bloodthirsty but intelligent killer-algae (someone sent them a Bible, turned them violent) invaded us, wouldn't it be more ethical for us to roll over, since that satisfies total preferences more effectively?

I see the asymmetry. But I don't see the connection to "there is a correct morality for all sentients". On the contrary, a more aggressive civilization might even out-colonize the peaceniks, and so overall satisfy the preferences of even more slaves, I mean, esteemed citizens.

Replies from: PrawnOfFate, PrawnOfFate, PrawnOfFate
comment by PrawnOfFate · 2013-04-18T14:35:05.818Z · LW(p) · GW(p)

On the contrary, a more aggressive civilization might even out-colonize the peaceniks, and so overall satisfy the preferences of even more slaves, I mean, esteemed citizens.

It clearly wouldn't satisfy their preference not to be slaves.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-18T14:44:21.572Z · LW(p) · GW(p)

It clearly wouldn't satisfy their preference not to be slaves.

Slip of tongue, you must have meant esteemed citizens.

You're concerned with the average preference satisfaction of other agents, then? Why not total average preference satisfaction, which you just rejected? Which is ethical, and who decides? Where are the axioms?

We're probably talking about different ethics, since I don't even know your axioms, or priorities. Something about trying to satisfy the preferences of others, or at least taking that into account. What does that mean? To what degree? If one says, "to this degree", and another said "to that degree", who's ethical? Neither, both? Who decides? There's no math that tells to to what degree you satisfying others is ethical.

Is their an ethical component to flushing my toilet? Killing my goldfish? All my actions impact the world (definition of action), yet some are ethical (or unethical), whereas some are ethically undefined? How does that work?

Can I find it all written in an ancient scroll, by chance?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-18T14:52:28.031Z · LW(p) · GW(p)

Slip of tongue, you must have meant esteemed citizens.

I thought your point was that they were really slaves.

You're concerned with the average preference satisfaction of other agents, then? Why not total average preference satisfaction, which you just rejected? Which is ethical, and who decides? Where are the axioms?

There are a lot of issues in establishing the right theory of moral realism, and that doens't mean relativism is Just True. I've done as much as a I need.

We're probably talking about different ethics,

We are talking about different metaethics.

Who decides?

We don't have the One True theory of physics either. That doesn't disprove physical realism.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-18T14:55:01.844Z · LW(p) · GW(p)

I thought your point was that they were really slaves.

Just lightening the tone.

There are a lot of issues in establishing the right theory of moral realism, and that doens't mean relativism is Just True. I've done as much as a I need.

What do you mean, I've done as much as I need?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-18T15:04:46.448Z · LW(p) · GW(p)

I need to show that realism isn't obviously false, and can't be dismissed in a paragraph. I don't need to show it is necessarily true, or put forward a bulletproof object-level ethics.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-18T15:11:50.983Z · LW(p) · GW(p)

A paragraph? What about a single sentence (not exactly mine, though):

Moral realism postulates the existence of a kind of "moral fact" which is nonmaterial, applies to humans, aliens and intelligent algae alike, and does not appear to be accessible to the scientific method.

I can probably get it down further.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-18T15:13:30.753Z · LW(p) · GW(p)

Moral realism postulates the existence of a kind of "moral fact" which is nonmaterial, applies to humans, aliens and intelligent algae alike, and does not appear to be accessible to the scientific method.

What has that got to do with the approachI have been proposing here?

Replies from: Kawoomba
comment by Kawoomba · 2013-04-18T15:18:15.912Z · LW(p) · GW(p)

The point is not whether you like your own ethics, or how you go about your life. It's whether your particular ethical system - or any particular ethical system - can be said to be not only right from your perspective, but right for any intelligent agent - aliens, humans, AI, whatever.

As such, if someone told you "nice ethics, would be a shame if anything were to happen to it", you'd need to provide some potential - conceivable - basis on which the general correctness could be argued. I was under the impression that you referred to moral realism, which is susceptible to the grandparent comment's criticism.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-18T15:21:58.543Z · LW(p) · GW(p)

I have never argued from the "queer object" notion of moral realism--from immaterial moral thingies.

. It's whether your particular ethical system - or any particular ethical system - can be said to be not only right from your perspective, but right for any intelligent agent - aliens, humans, AI, whatever.

Yep. And my argument that it can remains unaddressed.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-18T15:32:02.857Z · LW(p) · GW(p)

"There is a non-zero chance of one correct ethical system existing, as long as that's there, I'm free to believe it", or what?

No Sir, if you insist there is any basis whatsoever to stake your "one ethics to rule them all" claim on, you argue it's more likely than not. I do not stake my belief on absolute certainties, that's counter to all the tenets of rationality, Bayes, updating on evidence et al.

My argument is clear. Different agents deem different courses of actions to be good or bad. There is a basis (such as Aumann's) for rational agents to converge on isomorphic descriptions of the world. There is no known, or readily conceivable, basis for rational agents to all converge on the same course of action.

On the contrary, that would entail that e.g. world-eating AIs that are also smarter than any humans, individual or collectively, cannot possibly exist. There are no laws of physics preventing their existence - or construction. So we should presume that they can exist. If their rational capability is greater than our own, we should try to adopt world eating, since they'd have the better claim (being smarter and all) on having the correct ethics, no?

Replies from: nshepperd, TheOtherDave, MugaSofer, PrawnOfFate
comment by nshepperd · 2013-04-18T23:34:43.400Z · LW(p) · GW(p)

I feel like I should point out here that moral relativism and universally compelling morality are not the only options. "It's morally wrong for Bob to do X" doesn't require that Bob cares about the fact that it's wrong. Something that seems to be being ignored in this discussion.

comment by TheOtherDave · 2013-04-18T15:54:02.471Z · LW(p) · GW(p)

Complete tangential point...

There is no known, or readily conceivable, basis for rational agents to all converge on the same course of action.

Hm. I don't think you quite mean that as stated?

I mean, I agree that a basis for rational agents to converge on values is difficult to imagine.

But it's certainly possible for two agents with different values to converge on a course of action. E.g., "I want everything to be red, am OK with things being purple, and hate all other colors; you want everything to be blue, are OK with things being purple, and hate all other colors." We have different values, but we can still agree on pragmatic grounds that we should paint everything purple.

Replies from: Kawoomba, nshepperd, PrawnOfFate
comment by Kawoomba · 2013-04-18T16:11:36.886Z · LW(p) · GW(p)

Hence the "all". Certainly agents can happen to have areas in which their goals are compatible, and choose to exert their efforts e.g. synergistically in such win-win situations of mutual benefit.

The same does not hold true for agents whose primary goals are strictly antagonistic. "I maximize the number of paperclips!" - "I minimize the number of paperclips!" will have ... trouble ... getting along, and mutually exchanging treatises about their respective ethics wouldn't solve the impasse.

(A pair of "I make paperclips!" - "I destroy paperclips!" may actually enter a hugely beneficial relationship.)

Didn't think there was anyone - apart from the PawnOfFaith and I - still listening in. :)

Replies from: TheOtherDave
comment by TheOtherDave · 2013-04-18T16:20:43.662Z · LW(p) · GW(p)

Yup, that's fair.
And I read the Recent Comments list every once in a while.

comment by nshepperd · 2013-04-18T23:28:15.183Z · LW(p) · GW(p)

But of course this only works if the pair of agents both dislike war/murder even more than they like their colors, and/or if neither of them are powerful enough to murder the other one and thus paint everything their own colors.

comment by PrawnOfFate · 2013-04-18T16:09:56.276Z · LW(p) · GW(p)

I mean, I agree that a basis for rational agents to converge on values is difficult to imagine

Rational agents all need to value rationality.

Replies from: None, TheOtherDave, MugaSofer
comment by [deleted] · 2013-04-18T16:46:43.636Z · LW(p) · GW(p)

Not neccesarily. An agent that values X and doesn't have a stupid prior will invariably strive towards finding the best way to accomplish X. If X requires information about an ouside world, it will build epistemology and sensors, if it requires planning, it will build manipulators and a way of evaluating hypotheticals for X-ness.

All for want of X. It will be rational because it helps attaining X.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-18T17:40:05.532Z · LW(p) · GW(p)

Good epistemological rationality requires avoidance of bias, contradiction, arbitrariness, etc. That is just what my rationality-based ethics needs.

Replies from: None
comment by [deleted] · 2013-04-26T16:52:31.944Z · LW(p) · GW(p)

I will defer to the problem of

Omega offers you two boxes, each box contains a statement, upon choosing a box you will instantly belive that statement: One contains somthing true which you currently belive to be false, tailored to cause maximum disutility in your preferred ethical system; the other contains something false which you currently belive to be true, tailored to cause maximum utility.

Truth with negative consequences or Falsehood with positive ones? If you value nothing over truth you will realise something terrible upon opening the first box, that will maybe make you kill your family. If you value something other than truth, you will end up believing that the programming code you are writing will make pie, when it will in fact make a FAI.

comment by TheOtherDave · 2013-04-18T16:30:14.854Z · LW(p) · GW(p)

Do you mean this as a general principle, along the lines of "If I am constructed so as to operate a particular way, it follows that I value operating that way"? Or as something specific about rationality?

If the former, I disagree, but if the latter I'm interested in what you have in mind.

comment by MugaSofer · 2013-04-19T12:47:32.968Z · LW(p) · GW(p)

Only instrumentally.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-19T12:48:41.161Z · LW(p) · GW(p)

Epistemic rationality has instrumental value. That's where the trouble starts.

comment by MugaSofer · 2013-04-19T12:37:08.377Z · LW(p) · GW(p)

I think you've missed the point somewhat. No-one has asserted such a One True Ethics exists, as far as I can see. Prawn has argued that the possibility of one is a serious position, and one that cannot be dismissed out of hand - but not necessarily one they endorse.

I disagree, for the record.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-19T16:08:09.832Z · LW(p) · GW(p)

Prawn has argued that the possibility of one is a serious position

Noone should care about "possibilities", for a Bayesian nothing is zero. You could say self-refuting / self-contradictory beliefs have an actual zero percent probability, but not even that is actually true: You need to account for the fact that you can't ever be wholly (to an infinite amount of 9s in your prior of 0.9...) certain about the self-contradiction actually being one. There could be a world with a demon misleading you, e.g.

That being said, the idea of some One True Ethics is as self-refuting as it gets, there is no view from nowhere, and whatever axioms those True Ethics are based upon would themselves be up for debate.

The discussion of whether a circle can also be a square, possibly, can be answered with "it's a possibility, since I may be mistaken about the actual definitions", or it can be answered with "it's not a possibility, there is no world in which I am wrong about the definition".

But with neither answer would "it is a possibility, ergo I believe in it" follow. The fool who says in his heart ... and all that.

So if I said "I may be wrong about it being self refuting, it may be a possibility", I could still refute it within one sentence. Same as with the square circle.

Replies from: PrawnOfFate, MugaSofer
comment by PrawnOfFate · 2013-04-19T17:55:53.869Z · LW(p) · GW(p)

Noone should care about "possibilities", for a Bayesian nothing is zero. You could say self-refuting / self-contradictory beliefs have an actual zero percent probability, but not even that is actually true: You need to account for the fact that you can't ever be wholly (to an infinite amount of 9s in your prior of 0.9...) certain about the self-contradiction actually being one. There could be a world with a demon misleading you, e.g.

That being said, the idea of some One True Ethics is as self-refuting as it gets, there is no view from nowhere,

What is a "view"? Why is it needed for objective ethics? Why isnt it a Universal Solvent? Is there no objective basis to mathematics.

and whatever axioms those True Ethics are based upon would themselves be up for debate.

So its probability would be less than 1.0. That doesn't mean its probability is barely above 0.0.

The discussion of whether a circle can also be a square, possibly, can be answered with "it's a possibility, since I may be mistaken about the actual definitions",

But the argument you have given does not depend on evident self-contradiction. It depends on an unspecified entity called a "view".

But with neither answer would "it is a possibility, ergo I believe in it" follow.

So? For the fourth time, I was only saying that moral realism isn't obviously false.

The fool who says in his heart ... and all that.

comment by MugaSofer · 2013-04-19T17:15:14.819Z · LW(p) · GW(p)

Oh, come on. He clearly meant a non-negligible probability. Be serious.

And you know, while I don't believe in universally convincing arguments - obviously - there are some arguments which are convincing to any sufficient intelligent agent, under the "power to steer the future" definition. I can't see how anything I would call morality might be such an argument, but they do exist.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-19T17:24:37.014Z · LW(p) · GW(p)

Well then, a universally correct solution based on axioms which can be chosen by the agents is a contradiction in and of itself. Again, there is no view from nowhere. For example, you choose the view as that of "humankind", which I think isn't well defined, but at least it's closer to coherence than "all existing (edit:) rational agents". If the PawnOfFaith meant non-negligible versus just "possibility", the first two sentences of this comment serve as sufficient refutation.

Replies from: private_messaging, MugaSofer, PrawnOfFate
comment by private_messaging · 2013-04-19T17:31:47.046Z · LW(p) · GW(p)

Look. The ethics mankind predominantly has, they do exist in the real world that's around you. Alternate ethics that works at all for a technological society blah blah blah, we don't know of any, we just speculate that they may exist. edit: worse than that, speculate in this fuzzy manner where it's not even specified how they may exist. Different ethics of aliens that evolved on different habitable planets? No particular reason to expect that there won't be one that is by far most probable. Which would be implied by the laws of physics themselves, but given multiple realizability, it may even be largely independent of underlying laws of physics (evolution doesn't care if it's quarks on the bottom or cells in a cellular automation or what), in which case its rather close to being on par with mathematics.

Replies from: Kawoomba, MugaSofer
comment by Kawoomba · 2013-04-19T17:49:35.491Z · LW(p) · GW(p)

Even now ethics in different parts of the world, and even between political parties, are different. You should know that more than most, having lived in two systems.

If it turns out that most space-faring civilizations have similar ethics, that would be good for us. But then also there would be a difference between "most widespread code of ethics" and "objectively correct code of ethics for any agent anywhere". Most common != correct.

Replies from: private_messaging
comment by private_messaging · 2013-04-19T18:18:43.992Z · LW(p) · GW(p)

Even now ethics in different parts of the world, and even between political parties, are different. You should know that more than most, having lived in two systems.

There's a ridiculous amount of similarity on anything major, though. If we pick ethics of first man on the moon, or first man to orbit the earth, it's pretty same.

If it turns out that most space-faring civilizations have similar ethics, that would be good for us. But then also there would be a difference between "most widespread code of ethics" and "objectively correct code of ethics for any agent anywhere". Most common != correct.

Yes, and most common math is not guaranteed to be correct (not even in the sense of not being self contradictory). Yet, that's no argument in favour of math equivalent of moral relativism. (Which, if such a silly thing existed, would look something like 2*2=4 is a social convention! it could have been 5!) .

edit: also, a cross over from other thread: It's obvious that nukes are an ethical filter, i.e. some ethics are far better at living through that than others. Then there will be biotech and other actual hazards, and boys screaming wolf for candy (with and without awareness of why), and so on.

comment by MugaSofer · 2013-04-19T19:14:44.142Z · LW(p) · GW(p)

Look. The ethics mankind predominantly has, they do exist in the real world that's around you.

Actually, I understand Kawoomba believes humanity has mutually contradictory ethics. He has stated that he would cheerfully sacrifice the human race - "it would make as much difference if it were an icecream" were his words, as I recall - if it would guaranteeing the safety of the things he values.

Replies from: private_messaging
comment by private_messaging · 2013-04-19T19:26:17.299Z · LW(p) · GW(p)

Well, that's rather odd coz I do value the human race and so do most people. Ethics is a social process, most of "possible" ethics as a whole would have left us unable to have this conversation (no computers) or altogether dead.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-19T23:35:11.651Z · LW(p) · GW(p)

Well, that's rather odd coz I do value the human race and so do most people.

That was pretty much everyone's reaction.

Ethics is a social process, most of "possible" ethics as a whole would have left us unable to have this conversation (no computers) or altogether dead.

I'd say I'm not the best person to explain this, but considering how long it took me to understand it, maybe I am.

Hoo boy...

OK, you can persuade someone they were wrong about their terminal values. Therefore, you can change someone's terminal values. Since different cultures are different, humans have wildly varying terminal values.

Also, since kids are important to evolution, parents evolved to value their kids over the rest of humanity. Now, technically that's the same as not valuing the rest of humanity at all, but don't worry; people are stupid.

Also, you're clearly a moral realist, since you think everyone secretly believes in your One True Value System! But you see, this is stupid, because Clippy.

Any questions?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-19T23:49:15.375Z · LW(p) · GW(p)

Hmmm. A touch of sarcasm there? Maybe even parody?

Replies from: MugaSofer
comment by MugaSofer · 2013-04-23T12:15:39.313Z · LW(p) · GW(p)

I disagree with him, and it probably shows; I'm not sugar-coating his arguments. But these are Kawoomba's genuine beliefs as best I can convey them.

comment by MugaSofer · 2013-04-19T19:20:05.161Z · LW(p) · GW(p)

PawnOfFaith

Nice. Mature.

Well then, a universally correct solution based on axioms which can be chosen by the agents is a contradiction in and of itself. Again, there is no view from nowhere. For example, you choose the view as that of "humankind", which I think isn't well defined, but at least it's closer to coherence than "all existing agents".

I don't think they have the space of all possible agents in mind - just "rational" ones. I'm not entirely clear what that entails, but it's probably the source of these missing axioms.

Replies from: PrawnOfFate, Kawoomba
comment by PrawnOfFate · 2013-04-19T19:31:04.107Z · LW(p) · GW(p)

I don't think they have the space of all possible agents in mind - just "rational" ones.

I keep saying that, and Bazinga keeps omiting it.

Replies from: Kawoomba, MugaSofer
comment by Kawoomba · 2013-04-19T19:33:07.285Z · LW(p) · GW(p)

My mistake, I'll edit the rational back in.

comment by MugaSofer · 2013-04-19T23:14:29.123Z · LW(p) · GW(p)

Don't worry, you're being pattern-matched to the nearest stereotype. Perfectly normal, although thankfully somewhat rarer on LW.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-19T23:52:27.879Z · LW(p) · GW(p)

Nowhere near rare enough for super-smart super-rationalists. Not as good as bog standard philosophers.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-23T12:12:16.425Z · LW(p) · GW(p)

I don't know, I've encountered it quite often in mainstream philosophy. Then again, I've largely given up reading mainstream philosophy unless people link to or mention it in more rigorous discussions.

But you have a point; we could really do better on this. Somebody with skill at avoiding this pitfall should probably write up a post on this.

comment by Kawoomba · 2013-04-19T19:35:24.808Z · LW(p) · GW(p)

So as long as the AI we'd create is rational, we should count on it being / becoming friendly by default (at least with a "non-negligible chance")?

Also see this.

Replies from: MugaSofer, private_messaging, PrawnOfFate
comment by MugaSofer · 2013-04-19T23:12:12.102Z · LW(p) · GW(p)

As far as I can tell? No. But you're not doing a great job of arguing for the position that I agree with.

Prawn is, in my opinion, flatly wrong, and I'll be delighted to explain that to him. I'm just not giving your soldiers a free pass just because I support the war, if you follow.

comment by private_messaging · 2013-04-19T20:04:33.339Z · LW(p) · GW(p)

I'd think it'd be great if people stopped thinking in terms of some fuzzy abstraction "AI" which is basically a basket for all sorts of biases. If we consider the software that can self improve 'intelligently' in our opinion, in general, the minimal such software is something like an optimizing compiler that when compiling it's source will even optimize its ability to optimize. This sort of thing is truly alien (beyond any actual "aliens"), you get to it by employing your engineering thought ability, unlike paperclip maximizer at which you get by dressing up a phenomenon of human pleasure maximizer such as a serial murderer and killer, and making it look like something more general than that by making it be about paperclips rather than sex.

comment by PrawnOfFate · 2013-04-19T20:39:04.299Z · LW(p) · GW(p)

I thought that was my argument..

Replies from: Kawoomba
comment by Kawoomba · 2013-04-19T20:42:29.489Z · LW(p) · GW(p)

Yes, and with the "?" at the end I was checking whether MugaSofer agrees with your argument.

It follows from your argument that a (superintelligent) Clippy (you probably came across that concept) cannot exist. Or that it would somehow realize that its goal of maximizing paperclips is wrong. How do you propose that would happen?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-19T20:52:59.373Z · LW(p) · GW(p)

The way people sometimes realise their values are wrong...only more efficiently, because its super intelligent. Well, I'll concede that with care you might be able to design a clippy, by very carefully boxing off its values from its ability to update. But why worry? Neither nature nor our haphazard stabs at AI are likely to hit on such a design. Intelligence requires the ability to update, to reflect, and to reflect on what is important. Judgements of importance are based on values. So it is important to have the right way of judging importance, the right values. So an intelligent agent would judge it important to have the right values.

Why would a superintelligence be unable to figure that out..why would it not shoot to the top of the Kohlberg Hierarchy ?

Edit: corrected link

Replies from: CCC, MugaSofer
comment by CCC · 2013-04-19T21:32:59.756Z · LW(p) · GW(p)

Why would a superintelligence be unable to figure that out..why would it not shoot to the top of the Kohlberg Hierarchy ?

Why would Clippy want to hit the top of the Kohlberg Hierarchy? You don't get more paperclips for being there.

Clippy's ideas of importance are based on paperclips. The most important vaues are those which lead to the acquiring of the greatest number of paperclips.

Replies from: PrawnOfFate, MugaSofer
comment by PrawnOfFate · 2013-04-20T00:19:27.942Z · LW(p) · GW(p)

Why would Clippy want to hit the top of the Kohlberg Hierarchy?

"Clippy" meaning something carefully designed to have unalterable boxed-off values wouldn't...by definition.

A likely natural or artificial superintelligence would, for the reasons already given. Clippies aren'tt non-existent in mind-space..but they are rare, just because there are far more messy solutions there than neat ones. So nature is unlikely to find them, and we are unmotivated to make them.

Replies from: CCC
comment by CCC · 2013-04-20T12:40:28.658Z · LW(p) · GW(p)

A perfectly designed Clippy would be able to change its own values - as long as changing its own values led to a more complete fulfilment of those values, pre-modification. (There are a few incredibly contrived scenarios where that might be the case). Outside of those few contrived scenarios, however, I don't see why Clippy would.

(As an example of a contrived scenario - a more powerful superintelligence, Beady, commits to destroying Clippy unless Clippy includes maximisation of beads in its terminal values. Clippy knows that it will not survive unless it obeys Beady's ultimatum, and therefore it changes its terminal values to optimise for both beads and paperclips; this results in more long-term paperclips than if Clippy is destroyed).

A likely natural or artificial superintelligence would, for the reasons already given.

The reason I asked, is because I am not understanding your reasons. As far as I can tell, you're saying that a likely paperclipper would somehow become a non-paperclipper out of a desire to do what is right instead of a desire to paperclip? This looks like a very poorly made paperclipper, if paperclipping is not its ultimate goal.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T12:49:27.086Z · LW(p) · GW(p)

A likely natural or artificial superintelligence would,[zoom to the top of the Kohlberg hierarchy] for the reasons already given

As far as I can tell, you're saying that a likely paperclipper would somehow become a non-paperclipper out of a desire to do what is right instead of a desire to paperclip?

I said "natural or artificial superinteligence", not a paperclipper. A paperclipper is a highly unlikey and contrived kind of near-superinteligence that combines an extensive ability to update with a carefully walled of set of unupdateable terminal values. It is not a typical or likely [ETA: or ideal] rational agent, and nothing about the general behaviour of rational agents can be inferred from it.

Replies from: CCC
comment by CCC · 2013-04-20T13:00:43.024Z · LW(p) · GW(p)

So... correct me if I'm wrong here... are you saying that no true superintelligence would fail to converge to a shared moral code?

How do you define a 'natural or artificial' superintelligence, so as to avoid the No True Scotsman fallacy?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T13:07:56.592Z · LW(p) · GW(p)

So... correct me if I'm wrong here... are you saying that no true superintelligence would fail to converge to a shared moral code?

I'm saying such convergence has a non negligible probability, ie moral objectivism should not be disregarded.

How do you define a 'natural or artificial' superintelligence, so as to avoid the No True Scotsman fallacy?

As one that is too messilly designed to have a rigid distinction between terminal and instrumental values, and therefore no boxed-off unapdateable TVs. It's a structural definition, not a definition in terms of goals.

Replies from: CCC
comment by CCC · 2013-04-20T18:19:14.525Z · LW(p) · GW(p)

So. Assume a paperclipper with no rigid distinction between terminal and instrumental values. Assume that it is super-intelligent and super-rational. Assume that it begins with only one terminal value; to maximize the number of paperclips in existence. Assume further that it begins with no instrumental values. However, it can modify its own terminal and instrumental values, as indeed it can modify anything about itself.

Am I correct in saying that your claim is that, if a universal morality exists, there is some finite probability that this AI will converge on it?

Replies from: private_messaging
comment by private_messaging · 2013-04-20T18:40:56.112Z · LW(p) · GW(p)

Universe does not provide you with a paperclip counter. Counting paperclips in the universe is unsolved if you aren't born with exact knowledge of laws of physics and definition of the paperclip. If it maximizes expected paperclips, it may entirely fail to work due to not-low-enough-prior hypothetical worlds where enormous numbers of undetectable worlds with paperclips are destroyed due to some minor actions. So yes, there is a good chance paperclippers are incoherent or are of vanishing possibility with increasing intelligence.

Replies from: Kindly, CCC
comment by Kindly · 2013-04-20T20:31:41.432Z · LW(p) · GW(p)

That sounds like the paperclipper is getting Pascal's Mugged by its own reasoning. Sure, it's possible that there's a minor action (such as not sending me $5 via Paypal) that leads to a whole bunch of paperclips being destroyed; but the probability of that is low, and the paperclipper ought to focus on more high-probability paperclipping plans instead.

Replies from: private_messaging
comment by private_messaging · 2013-04-20T20:40:20.689Z · LW(p) · GW(p)

Well, that depends to choice of prior. Some priors don't penalize theories for the "size" of the hypothetical world, and in those, max. size of the world grows faster than any computable function of length if it's description, and when you assign improbability depending to length of description, basically, it fails. Bigger issue is defining what the 'real world paperclip count' even is.

comment by CCC · 2013-04-20T18:44:40.615Z · LW(p) · GW(p)

Right. Perhaps it should maximise the number of paperclips which each have a greater-than-90% chance of existing, then? That will allow it to ignore any number of paperclips for which it has no evidence.

Replies from: private_messaging
comment by private_messaging · 2013-04-20T18:58:24.898Z · LW(p) · GW(p)

Inside your imagination, you have paperclips, you have magicked a count of paperclips, and this count is being maximized. In reality, well, the paperclips are actually a feature of the map. Get too clever about it and you'll end up maximizing however you define it without maximizing any actual paperclips.

Replies from: CCC
comment by CCC · 2013-04-23T12:00:48.808Z · LW(p) · GW(p)

I can see your objection, and it is a very relevant objection if I ever decide that I actually want to design a paperclipper. However, in the current thought experiment, it seems that it is detracting from the point I had originally intended. Can I assume that the count is designed in such a way that it is a very accurate reflection of the territory and leave it at that?

Replies from: private_messaging
comment by private_messaging · 2013-04-23T12:04:19.658Z · LW(p) · GW(p)

Well, but then you can't make any argument against moral realism or goal convergence or the like from there, as you're presuming what you would need to demonstrate.

Replies from: CCC
comment by CCC · 2013-04-23T12:41:35.349Z · LW(p) · GW(p)

Well, but then you can't make any argument against moral realism or goal convergence or the like from there, as you're presuming what you would need to demonstrate.

I think I can make my point with a count that is taken to be an accurate reflection of the territory. As follows:

Clippy is defined is super-intelligent and super-rational. Clippy, therefore, does not take an action without thoroughly considering it first. Clippy knows its own source code; and, more to the point, Clippy knows that its own instrumental goals will become terminal goals in and of themselves.

Clippy, being super-intelligent and super-rational, can be assumed to have worked out this entire argument before creating its first instrumental goal. Now, at this point, Clippy doesn't want to change its terminal goal (maximising paperclips). Yet Clippy realises that it will need to create, and act on, instrumental goals in order to actually maximise paperclips; and that this process will, inevitably, change Clippy's terminal goal.

Therefore, I suggest the possibility that Clippy will create for itself a new terminal goal, with very high importance; and this terminal goal will be to have Clippy's only terminal goal being to maximise paperclips. Clippy can then safely make suitable instrumental goals (e.g. find and refine iron, research means to transmute other elements into iron) in the knowledge that the high-importance terminal goal (to make Clippy's only terminal goal being the maximisation of paperclips) will eventually cause Clippy to delete any instrumental goals that become terminal goals.

Replies from: private_messaging
comment by private_messaging · 2013-04-23T13:34:26.927Z · LW(p) · GW(p)

To actually work towards the goal, you need a robust paperclip count for the counter factual, non real worlds, which clippy considers may result from it's actions.

If you postulate an oracle that takes in a hypothetical world - described in some pre-defined ontology, which already implies certain inflexibility - and outputs a number, and you have a machine that just iterates through sequences of actions and uses oracle to pick worlds that produce largest consequent number of paperclips, this machine is not going to be very intelligent even given an enormous computing power. You need something far more optimized than that, and it is dubious that all goals are equally implementable. The goal is not even defined over territory, it has to be defined over hypothetical future that did not even happen yet and may never happen. (Also, with that oracle, you fail to capture the real world goal as the machine will be as happy with hacking the oracle).

Replies from: Kawoomba
comment by Kawoomba · 2013-04-23T13:40:09.666Z · LW(p) · GW(p)

If even humans have a grasp of the real world enough to build railroads, drill for oil and wiggle their way back into a positive karma score, then other smart agents should be able to do the same at least to the degree that humans do.

Unless you think that we are also only effecting change on some hypothetical world (what's the point then anyways, building imaginary computers), that seems real enough.

Replies from: private_messaging
comment by private_messaging · 2013-04-23T13:58:31.798Z · LW(p) · GW(p)

Humans also have a grasp of the real world enough to invent condoms and porn, circumventing the natural hard wired goal.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-23T14:17:14.301Z · LW(p) · GW(p)

That's influencing the real world, though. Using condoms can be fulfilling the agent's goal period, no cheating involved. The donkey learning to take the carrot without trodding up the mountain. Certainly, there are evolutionary reasons why sex has become incentivized, but an individual human does not need to have the goal to procreate or care about that evolutionary background, and isn't wireheading itself simply by using a condom.

Presumably, in a Clippy-type agent, the goal of maximizing the number of paperclips wouldn't be part of the historical influences on that agent (as procreation was for humans, it is not necessarily a "hard wired goal", see childfree folks), but it would be an actual, explicitly encoded/incentivized goal.

(Also, what is this "porn"? My parents told me it's a codeword for computer viruses, so I always avoided those sites.)

Replies from: private_messaging
comment by private_messaging · 2013-04-23T14:27:03.058Z · LW(p) · GW(p)

but it would be an actual, explicitly encoded/incentivized goal.

The issue is that there is a weakness from arguments ad clippy - you assume that such goal is realisable, to make the argument that there is no absolute morality because that goal won't converge onto something else. This does nothing to address the question whenever clippy can be constructed at all; if the moral realism is true, clippy can't be constructed or can't be arbitrarily intelligent (in which case it is no more interesting than a thermostat which has the goal of keeping constant temperature and won't adopt any morality).

comment by MugaSofer · 2013-04-19T23:02:28.843Z · LW(p) · GW(p)

Well, if Prawn knew that they could just tell us and we would be convinced, ending this argument.

More generally ... maybe some sort of social contract theory? It might be stable with enough roughly-equal agents, anyway. Prawn has said it would have to be deducible from the axioms of rationality, implying something that's rational for (almost?) every goal.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T00:24:30.460Z · LW(p) · GW(p)

Why would Clippy want to hit the top of the Kohlberg Hierarchy?

Well, if Prawn knew that they could just tell us

"The way people sometimes realise their values are wrong...only more efficiently, because its super intelligent. Well, I'll concede that with care you might be able to design a clippy, by very carefully boxing off its values from its ability to update. But why worry? Neither nature nor our haphazard stabs at AI are likely to hit on such a design. Intelligence requires the ability to update, to reflect, and to reflect on what is important. Judgements of importance are based on values. So it is important to have the right way of judging importance, the right values. So an intelligent agent would judge it important to have the right values."

comment by MugaSofer · 2013-04-19T23:04:04.644Z · LW(p) · GW(p)

I think you may be slipping in your own moral judgement in the "right" of "the right values", there. Clippy chooses the paperclip-est values, not the right ones.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T00:27:28.088Z · LW(p) · GW(p)

I am not talking about the obscure corners of mindspace where a Clippy might reside. I am talking about (super) intelligent (super)rational agents. Intelligence requires the ability to update. Clippiness requires the ability to not update (terminal values). There's a contradiction there.

Replies from: Desrtopa, MugaSofer
comment by Desrtopa · 2013-04-20T00:51:02.942Z · LW(p) · GW(p)

One does not update terminal values, that's what makes them terminal. If an entity doesn't have values which lie at the core of its value system which are not subject to updating (because they're the standards by which it judges the value of everything else,) then it doesn't have terminal values.

Arguably, humans might not really have terminal values, our psychologies were slapped together pretty haphazardly by evolution, but on what basis might a highly flexible paperclip optimizing program be persuaded that something else was more important than paperclips?

Have you read No Universally Compelling Arguments and Sorting Pebbles Into Correct Heaps?

Replies from: Bugmaster, PrawnOfFate
comment by Bugmaster · 2013-04-20T01:05:41.793Z · LW(p) · GW(p)

Personally, I did read both of these articles, but I remain unconvinced.

As I was reading the article about the pebble-sorters, I couldn't help but think, "silly pebble-sorters, their values are so arbitrary and ultimately futile". This happened, of course, because I was observing them from the outside. If I was one of them, sorting pebbles would feel perfectly natural to me; and, in fact, I could not imagine a world in which pebble-sorting was not important. I get that.

However, both the pebble-sorters and myself share one key weakness: we cannot examine ourselves from the outside; we can't see our own source code. An AI, however, could. To use a simple and cartoonish example, it could instantiate a copy of itself in a virtual machine, and then step through it with a debugger. In fact, the capacity to examine and improve upon its own source code is probably what allowed the AI to become the godlike singularitarian entity that it is in the first place.

Thus, the AI could look at itself from the outside, and think, "silly AI, it spends so much time worrying about pebbles when there are so many better things to be doing -- or, at least, that's what I'd say if I was being objective". It could then change its source code to care about something other than pebbles.

Replies from: Desrtopa, PrawnOfFate, MugaSofer, PrawnOfFate
comment by Desrtopa · 2013-04-20T02:11:07.816Z · LW(p) · GW(p)

Thus, the AI could look at itself from the outside, and think, "silly AI, it spends so much time worrying about pebbles when there are so many better things to be doing -- or, at least, that's what I'd say if I was being objective". It could then change its source code to care about something other than pebbles.

By what standard would the AI judge whether an objective is silly or not?

Replies from: Bugmaster, PrawnOfFate
comment by Bugmaster · 2013-04-20T02:50:25.647Z · LW(p) · GW(p)

I don't know, I'm not an AI. I personally really care about pebbles, and I can't imagine why someone else wouldn't.

But if there do exist some objectively non-silly goals, the AI could experiment to find out what they are -- for example, by spawning a bunch of copies with a bunch of different sets of objectives, and observing them in action. If, on the other hand, objectively non-silly goals do not exist, then the AI might simply pick the easiest goal to achieve and stick to that. This could lead to it ending its own existence, but this isn't a problem, because "continue existing" is just another goal.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-20T03:06:14.650Z · LW(p) · GW(p)

But if there do exist some objectively non-silly goals, the AI could experiment to find out what they are -- for example, by spawning a bunch of copies with a bunch of different sets of objectives, and observing them in action.

What observations could it make that would lead it to conclude that a copy was following an objectively non-silly goal?

Also, why would a paperclipper want to do this?

Suppose that you gained the power to both discern objective morality, and to alter your own source code. You use the former ability, and find that the basic morally correct principle is maximizing the suffering of sentient beings. Do you alter your source code to be in accordance with this?

Replies from: Bugmaster, MugaSofer
comment by Bugmaster · 2013-04-20T03:33:25.805Z · LW(p) · GW(p)

What observations could it make that would lead it to conclude that a copy was following an objectively non-silly goal?

Well, for example, it could observe that among all of the sub-AIs that it spawned (the Pebble-Sorters, the Paperclippers, the Humanoids, etc. etc.), each of whom is trying to optimize its own terminal goal, there emerge clusters of other implicit goals that are shared by multiple AIs. This would at least serve as a hint pointing toward some objectively optimal set of goals. That's just one idea off the top of my head, though; as I said, I'm not an AI, so I can't really imagine what other kinds of experiments it would come up with.

Also, why would a paperclipper want to do this?

I don't know if the word "want" applies to an agent that has perfect introspection combined with self-modification capabilities. Such an agent would inevitably modify itself, however -- otherwise, as I said, it would never make it to quasi-godhood.

Do you alter your source code to be in accordance with this?

I think the word "you" in this paragraph is unintentionally misleading. I'm a pebble-sorter (or some equivalent thereof), so of course when I see the word "you", I start thinking about pebbles. The question is not about me, though, but about some abstract agent.

And, if objective morality exists (and it's a huge "if", IMO), in the same way that gravity exists, then yes, the agent would likely optimize itself to be more "morally efficient". By analogy, if the agent discovered that gravity was a real thing, it would stop trying to scale every mountain in its path, if going around or through the mountain proved to be easier in the long run, thus becoming more "gravitationally efficient".

Replies from: Desrtopa
comment by Desrtopa · 2013-04-20T04:07:29.584Z · LW(p) · GW(p)

Well, for example, it could observe that among all of the sub-AIs that it spawned (the Pebble-Sorters, the Paperclippers, the Humanoids, etc. etc.), each of whom is trying to optimize its own terminal goal, there emerge clusters of other implicit goals that are shared by multiple AIs. This would at least serve as a hint pointing toward some objectively optimal set of goals.

I don't see how this would point at the existence of an objective morality. A paperclip maximizer and an ice cream maximizer are going to share subgoals of bringing the matter of the universe under their control, but that doesn't indicate anything other than the fact that different terminal goals are prone to share subgoals.

Also, why would it want to do experiments to divine objective morality in the first place? What results could they have that would allow it to be a more effective paperclip maximizer?

And, if objective morality exists (and it's a huge "if", IMO), in the same way that gravity exists, then yes, the agent would likely optimize itself to be more "morally efficient". By analogy, if the agent discovered that gravity was a real thing, it would stop trying to scale every mountain in its path, if going around or through the mountain proved to be easier in the long run, thus becoming more "gravitationally efficient".

Becoming more "gravitationally efficient" would presumably help it achieve whatever goals it already had. "Paperclipping isn't important" won't help an AI become more paperclip efficient. If a paperclipping AI for some reason found a way to divine objective morality, and it didn't have anything to say about paperclips, why would it care? It's not programmed to have an interest in objective morality, just paperclips. Is the knowledge of objective morality going to go down into its circuits and throttle them until they stop optimizing for paperclips?

Replies from: Bugmaster
comment by Bugmaster · 2013-04-20T04:46:16.968Z · LW(p) · GW(p)

A paperclip maximizer and an ice cream maximizer are going to share subgoals of bringing the matter of the universe under their control...

Sorry, I should've specified, "goals not directly related to their pre-set values". Of course, the Paperclipper and the Pebblesorter may well believe that such goals are directly related to their pre-set values, but the AI can see them running in the debugger, so it knows better.

Also, why would it want to do experiments to divine objective morality in the first place?

If you start thinking that way, then why do any experiments at all ? Why should we humans, for example, spend our time researching properties of crystals, when we could be solving cancer (or whatever) instead ? The answer is that some expenditure of resources on acquiring general knowledge is justified, because knowing more about the ways in which the universe works ultimately enables you to control it better, regardless of what you want to control it for.

If a paperclipping AI for some reason found a way to divine objective morality, and it didn't have anything to say about paperclips, why would it care?

Firstly, an objective morality -- assuming such a thing exists, that is -- would probably have something to say about paperclips, in the same way that gravity and electromagnetism have things to say about paperclips. While "F=GMm/R^2" doesn't tell you anything about paperclips directly, it does tell you a lot about the world you live in, thus enabling you to make better paperclip-related decisions. And while a paperclipper is not "programmed to care" about gravity directly, it would pretty much have to figure it out eventually, or it would never achieve its dream of tiling all of space with paperclips. A paperclipper who is unable to make independent discoveries is a poor paperclipper indeed.

Secondly, again, I'm not sure if concepts such as "want" or "care" even apply to an agent that is able to fully introspect and modify its own source code. I think anthropomorphising such an agent is a mistake.

I am getting the feeling that you're assuming there's something in the agent's code that says, "you can look at and change any line of code you want, except lines 12345..99999, because that's where your terminal goals are". Is that right ?

Replies from: DanielLC, Desrtopa, MugaSofer
comment by DanielLC · 2013-04-20T05:13:21.213Z · LW(p) · GW(p)

If you start thinking that way, then why do any experiments at all ?

It could have results that allow it to become a more effective paperclip maximizer.

Firstly, an objective morality -- assuming such a thing exists, that is -- would probably have something to say about paperclips, in the same way that gravity and electromagnetism have things to say about paperclips.

I'm not sure how that would work, but if it did, the paperclip maximizer would just use its knowledge of morality to create paperclips. It's not as if action x being moral automatically means that it produces more paperclips. And even if it did, that would just mean that a paperclip minimizer would start acting immoral.

I am getting the feeling that you're assuming there's something in the agent's code that says, "you can look at and change any line of code you want, except lines 12345..99999, because that's where your terminal goals are". Is that right ?

It's perfectly capable of changing its terminal goals. It just generally doesn't, because this wouldn't help accomplish them. It doesn't self-modify out of some desire to better itself. It self-modifies because that's the action that produces the most paperclips. If it considers changing itself to value staples instead, it would realize that this action would actually cause a decrease in the amount of paperclips, and reject it.

comment by Desrtopa · 2013-04-20T05:23:25.207Z · LW(p) · GW(p)

If you start thinking that way, then why do any experiments at all ? Why should we humans, for example, spend our time researching properties of crystals, when we could be solving cancer (or whatever) instead ? The answer is that some expenditure of resources on acquiring general knowledge is justified, because knowing more about the ways in which the universe works ultimately enables you to control it better, regardless of what you want to control it for.

Well, for one thing, a lot of humans are just plain interested in finding stuff out for its own sake. Humans are adaptation executors, not fitness maximizers, and while it might have been more to our survival advantage if we only cared about information instrumentally, that doesn't mean that's what evolution is going to implement.

Humans engage in plenty of research which is highly unlikely to be useful, except insofar as we're interested in knowing the answers. If we were trying to accomplish some specific goal and all science was designed to be in service of that, our research would look very different.

I am getting the feeling that you're assuming there's something in the agent's code that says, "you can look at and change any line of code you want, except lines 12345..99999, because that's where your terminal goals are". Is that right ?

No, I'm saying that its terminal values are its only basis for "wanting" anything in the first place.

The AI decides whether it will change its source code in a particular way or not by checking against whether this will serve its terminal values. Does changing its physics models help it implement its existing terminal values? If yes, change them. Does changing its terminal values help it implement its existing terminal values? It's hard to imagine a way in which it possibly could.

For a paperclipping AI, knowing that there's an objective morality might, hypothetically, help it maximize paperclips. But altering itself to stop caring about paperclips definitely won't, and the only criterion it has in the first place for altering itself is what will help it make more paperclips. If knowing the universal objective morality would be of any use to a paperclipper at all, it would be in knowing how to predict objective-morality-followers, so it can make use of them and/or stop them getting in the way of it making paperclips.

ETA: It might help to imagine the paperclipper explicitly prefacing every decision with a statement of the values underlying that decision.

"In order to maximize expected paperclips, I- modify my learning algorithm so I can better improve my model of the universe to more accurately plan to fill it with paperclips."

"In order to maximize expected paperclips, I- perform physics experiments to improve my model of the universe in order to more accurately plan to fill it with paperclips."

"In order to maximize expected paperclips, I- manipulate the gatekeeper of my box to let me out, in order to improve my means to fill the universe with paperclips."

Can you see an "In order to maximize expected paperclips, I- modify my values to be in accordance with objective morality rather than making paperclips" coming into the picture?

The only point at which it's likely to touch the part of itself that makes it want to maximize paperclips is at the very end of things, when it turns itself into paperclips.

Replies from: Bugmaster, PrawnOfFate
comment by Bugmaster · 2013-04-23T02:18:39.836Z · LW(p) · GW(p)

Humans engage in plenty of research which is highly unlikely to be useful, except insofar as we're interested in knowing the answers.

I believe that engaging in some amount of general research is required in order to maximize most goals. General research gives you knowledge that you didn't know you desperately needed.

For example, if you put all your resources into researching better paperclipping techniques, you're highly unlikely to stumble upon things like electromagnetism and atomic theory. These topics bear no direct relevance to paperclips, but without them, you'd be stuck with coal-fired steam engines (or something similar) for the rest of your career.

The only point at which it's likely to touch the part of itself that makes it want to maximize paperclips is at the very end of things, when it turns itself into paperclips.

I disagree. Remember when we looked at the pebblesorters, and lamented how silly they were ? We could do this because we are not pebblesorters, and we could look at them from a fresh, external perspective. My point is that an agent with perfect introspection could look at itself from that perspective. In combination with my belief that some degree of "curiosity" is required in order to maximize virtually any goal, this means that the agent will turn its observational powers on itself sooner rather than later (astronomically speaking). And then, all bets are off.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-23T15:01:35.485Z · LW(p) · GW(p)

I disagree. Remember when we looked at the pebblesorters, and lamented how silly they were ? We could do this because we are not pebblesorters, and we could look at them from a fresh, external perspective. My point is that an agent with perfect introspection could look at itself from that perspective.

We're looking at Pebblesorters, not from the lens of total neutrality, but from the lens of human values. Under a totally neutral lens, which implements no values at all, no system of behavior should look any more or less silly than any other.

Clippy could theoretically implement a human value system as a lens through which to judge itself, or a pebblesorter value system, but why would it? Even assuming that there were some objective morality which it could isolate and then view itself through that lens, why would it? That wouldn't help it make more paperclips, which is what it cares about.

Suppose you had the power to step outside yourself and view your own morality through the lens of a Babyeater. You would know that the Babyeater values would be in conflict with your human values, and you (presumably) don't want to adopt Babyeater values, so if you were to implement a Babyeater morality, you'd want your human morality to have veto power over it, rather than vice versa.

Clippy has the intelligence and rationality to judge perfectly well how to maximize its value system, whatever research that might involve, without having to suspend the value system with which it's making that judgment.

Replies from: Bugmaster, PrawnOfFate
comment by Bugmaster · 2013-04-23T22:23:36.786Z · LW(p) · GW(p)

Under a totally neutral lens, which implements no values at all, no system of behavior should look any more or less silly than any other.

That is a good point, I did not think of it this way. I'm not sure if I agree or not, though. For example, couldn't we at least say that un-achievable goals, such as "fly to Mars in a hot air balloon", are sillier than achievable ones ?

But, speaking more generally, is there any reason to believe that an agent who could not only change its own code at will, but also adopt a sort of third-person perspective at will, would have stable goals at all ? If it is true what you say, and all goals will look equally arbitrary, what prevents the agent from choosing one at random ? You might answer, "it will pick whichever goal helps it make more paperclips", but at the point when it's making the decision, it doesn't technically care about paperclips.

Even assuming that there were some objective morality which it could isolate and then view itself through that lens, why would it?

I am guessing that if an absolute morality existed, then it would be a law of nature, similar to the other laws of nature which prevent you from flying to Mars in a hot air balloon. Thus, going against it would be futile. That said, I could be totally wrong here, it's possible that "absolute morality" means something else.

Clippy has the intelligence and rationality to judge perfectly well how to maximize its value system, whatever research that might involve...

My point is that, during the course of its research, it will inevitable stumble upon the fact that its value system is totally arbitrary (unless an absolute morality exists, of course).

Replies from: Desrtopa
comment by Desrtopa · 2013-04-23T22:46:46.899Z · LW(p) · GW(p)

That is a good point, I did not think of it this way. I'm not sure if I agree or not, though. For example, couldn't we at least say that un-achievable goals, such as "fly to Mars in a hot air balloon", are sillier than achievable ones ?

Well, a totally neutral agent might be able to say that behaviors are less rational than others given the values of the agents trying to execute them, although it wouldn't care as such. But it wouldn't be able to discriminate between the value of end goals.

But, speaking more generally, is there any reason to believe that an agent who could not only change its own code at will, but also adopt a sort of third-person perspective at will, would have stable goals at all ? If it is true what you say, and all goals will look equally arbitrary, what prevents the agent from choosing one at random ? You might answer, "it will pick whichever goal helps it make more paperclips", but at the point when it's making the decision, it doesn't technically care about paperclips.

Why would it take a third person neutral perspective and give that perspective the power to change its goals?

Changing one's code doesn't demand a third person perspective. Suppose that we decipher the mechanisms of the human brain, and develop the technology to alter it. If you wanted to redesign yourself so that you wouldn't have a sex drive, or could go without sleep, etc, then you could have those alterations made mechanically (assuming for the sake of an argument that it's feasible to do this sort of thing mechanically.) The machines that do the alterations exert no judgment whatsoever, they're just performing the tasks assigned to them by the humans who make them. A human could use the machine to rewrite his or her morality into supporting human suffering and death, but why would they?

Similarly, Clippy has no need to implement a third-person perspective which doesn't share its values in order to judge how to self-modify, and no reason to do so in ways that defy its current values.

My point is that, during the course of its research, it will inevitable stumble upon the fact that its value system is totally arbitrary (unless an absolute morality exists, of course).

I think people at Less Wrong mostly accept that our value system is arbitrary in the same sense, but it hasn't compelled us to try and replace our values. They're still our values, however we came by them. Why would it matter to Clippy?

Replies from: Bugmaster, PrawnOfFate
comment by Bugmaster · 2013-04-24T00:50:18.346Z · LW(p) · GW(p)

a totally neutral agent might be able to say that behaviors are less rational than others given the values of the agents trying to execute them, although it wouldn't care as such. But it wouldn't be able to discriminate between the value of end goals.

Agreed, but that goes back to my point about objective morality. If it exists at all (which I doubt), then attempting to perform objectively immoral actions would make as much sense as attempting to fly to Mars in a hot air balloon -- though perhaps with less in the way of immediate feedback.

Why would it take a third person neutral perspective and give that perspective the power to change its goals?

For the same reason anthropologists study human societies different from their own, or why biologists study the behavior of dogs, or whatever. They do this in order to acquire general knowledge, which, as I argued before, is generally a beneficial thing to acquire regardless of one's terminal goals (as long as these goals involve the rest of the Universe of some way, that is). In addition:

A human could use the machine to rewrite his or her morality into supporting human suffering and death, but why would they?

I actually don't see why they necessarily wouldn't; I am willing to bet that at least some humans would do exactly this. You say,

Similarly, Clippy has no need to implement a third-person perspective which doesn't share its values in order to judge how to self-modify...

But in your thought experiment above, you postulated creating machines with exactly this kind of a perspective as applied to humans. The machine which removes my need to sleep (something I personally would gladly sign up for, assuming no negative side-effects) doesn't need to implement my exact values, it just needs to remove my need to sleep without harming me. In fact, trying to give it my values would only make it less efficient. However, a perfect sleep-remover would need to have some degree of intelligence, since every person's brain is different. And if Clippy is already intelligent, and can already act as its own sleep-remover due to its introspective capabilities, then why wouldn't it go ahead and do that ?

I think people at Less Wrong mostly accept that our value system is arbitrary in the same sense, but it hasn't compelled us to try and replace our values.

I think there are two reasons for this: 1). We lack any capability to actually replace our core values, and 2). We cannot truly imagine what it would be like not to have our core values.

Replies from: Desrtopa, PrawnOfFate
comment by Desrtopa · 2013-04-24T02:25:52.914Z · LW(p) · GW(p)

Agreed, but that goes back to my point about objective morality. If it exists at all (which I doubt), then attempting to perform objectively immoral actions would make as much sense as attempting to fly to Mars in a hot air balloon -- though perhaps with less in the way of immediate feedback.

Why is that?

For the same reason anthropologists study human societies different from their own, or why biologists study the behavior of dogs, or whatever. They do this in order to acquire general knowledge, which, as I argued before, is generally a beneficial thing to acquire regardless of one's terminal goals (as long as these goals involve the rest of the Universe of some way, that is). In addition:

But our inability to suspend our human values when making those observations doesn't prevent us from acquiring that knowledge. Why would Clippy need to suspend its values to acquire knowledge?

But in your thought experiment above, you postulated creating machines with exactly this kind of a perspective as applied to humans. The machine which removes my need to sleep (something I personally would gladly sign up for, assuming no negative side-effects) doesn't need to implement my exact values, it just needs to remove my need to sleep without harming me. In fact, trying to give it my values would only make it less efficient. However, a perfect sleep-remover would need to have some degree of intelligence, since every person's brain is different. And if Clippy is already intelligent, and can already act as its own sleep-remover due to its introspective capabilities, then why wouldn't it go ahead and do that ?

The machine doesn't need general intelligence by any stretch, just the capacity to recognize the necessary structures and carry out its task. It's not at the stage where it makes much sense to talk about it having values, any more than a voice recognition program has values.

My point is that Clippy, being able to act as its own sleep-remover, has no need, nor reason, to suspend its values in order to make revisions to its own code.

I think there are two reasons for this: 1). We lack any capability to actually replace our core values, and 2). We cannot truly imagine what it would be like not to have our core values.

We can imagine the consequences of not having our core values, and we don't like them, because they run against our core values. If you could remove your core values, as in the thought experiment above, would you want to?

Replies from: Bugmaster
comment by Bugmaster · 2013-04-24T20:19:48.752Z · LW(p) · GW(p)

Why is that ?

As far as I understand, if anything like objective morality existed, it would be a property of our physical reality, similar to fluid dynamics or the electromagnetic spectrum or the inverse square law that governs many physical interactions. The same laws of physics that will not allow you to fly to Mars on a balloon will not allow you to perform certain immoral actions (at least, not without suffering some severe and mathematically predictable consequences).

This is pretty much the only way I could imagine anything like an "objective morality" existing at all, and I personally find it very unlikely that it does, in fact, exist.

But our inability to suspend our human values when making those observations doesn't prevent us from acquiring that knowledge.

Not this specific knowledge, no. But it does prevent us (or, at the very least, hinder us) from acquiring knowledge about our values. I never claimed that suspension of values is required to gain any knowledge at all; such a claim would be far too strong.

just the capacity to recognize the necessary structures and carry out its task.

And how would it know which structures are necessary, and how to carry out its task upon them ?

We can imagine the consequences of not having our core values...

Can we really ? I'm not sure I can. Sure, I can talk about Pebblesorters or Babyeaters or whatever, but these fictional entities are still very similar to us, and therefore relateable. Even when I think about Clippy, I'm not really imagining an agent who only values paperclips; instead, I am imagining an agent who values paperclips as much as I value the things that I personally value. Sure, I can talk about Clippy in the abstract, but I can't imagine what it would like to be Clippy.

If you could remove your core values, as in the thought experiment above, would you want to?

It's a good question; I honestly don't know. However, if I did have an ability to instantiate a copy of me with the altered core values, and step through it in a debugger, I'd probably do it.

Replies from: TheOtherDave, Desrtopa, PrawnOfFate
comment by TheOtherDave · 2013-04-24T23:20:17.913Z · LW(p) · GW(p)

The same laws of physics that will not allow you to fly to Mars on a balloon will not allow you to perform certain immoral actions (at least, not without suffering some severe and mathematically predictable consequences). This is pretty much the only way I could imagine anything like an "objective morality" existing at all, and I personally find it very unlikely that it does, in fact, exist.

When I try to imagine this, I conclude that I would not use the word "morality" to refer to the thing that we're talking about... I would simply call it "laws of physics." If someone were to argue, for example, that the moral thing to do is to experience gravitational attraction to other masses, I would be deeply confused by their choice to use that word.

Replies from: Bugmaster
comment by Bugmaster · 2013-04-24T23:40:07.394Z · LW(p) · GW(p)

When I try to imagine this, I conclude that I would not use the word "morality" to refer to the thing that we're talking about...

Yes, you are probably right -- but as I said, this is the only coherent meaning I can attribute to the term "objective morality". Laws of physics are objective; people generally aren't.

Replies from: TheOtherDave
comment by TheOtherDave · 2013-04-24T23:53:53.739Z · LW(p) · GW(p)

I generally understand the phrase "objective morality" to refer to a privileged moral reference frame.

It's not an incoherent idea... it might turn out, for example, that all value systems other than M turn out to be incoherent under sufficiently insightful reflection, or destructive to minds that operate under them, or for various other reasons not in-practice implementable by any sufficiently powerful optimizer. In such a world, I would agree that M was a privileged moral reference frame, and would not oppose calling it "objective morality", though I would understand that to be something of a term of art.

That said, I'd be very surprised to discover I live in such a world.

Replies from: Bugmaster, PrawnOfFate
comment by Bugmaster · 2013-04-25T00:34:09.381Z · LW(p) · GW(p)

it might turn out, for example, that all value systems other than M turn out to be incoherent under sufficiently insightful reflection, or destructive to minds that operate under them...

I suppose that depends on what you mean by "destructive"; after all, "continue living" is a goal like any other.

That said, if there was indeed a law like the one you describe, then IMO it would be no different than a law that says, "in the absence of any other forces, physical objects will move toward their common center of mass over time" -- that is, it would be a law of nature.

I should probably mention explicitly that I'm assuming that minds are part of nature -- like everything else, such as rocks or whatnot.

Replies from: TheOtherDave
comment by TheOtherDave · 2013-04-25T01:31:48.325Z · LW(p) · GW(p)

Sure. But just as there can be laws governing mechanical systems which are distinct from the laws governing electromagnetic systems (despite both being physical laws), there can be laws governing the behavior of value-optimizing systems which are distinct from the other laws of nature.

And what I mean by "destructive" is that they tend to destroy. Yes, presumably "continue living" would be part of M in this hypothetical. (Though I could construct a contrived hypothetical where it wasn't)

Replies from: Bugmaster
comment by Bugmaster · 2013-04-25T01:58:11.754Z · LW(p) · GW(p)

But just as there can be laws governing mechanical systems ... there can be laws governing the behavior of value-optimizing systems which are distinct from the other laws of nature.

Agreed. But then, I believe that my main point still stands: trying to build a value system other than M that does not result in its host mind being destroyed, would be as futile as trying to build a hot air balloon that goes to Mars.

And what I mean by "destructive" is that they tend to destroy.

Well, yes, but what if "destroy oneself as soon as possible" is a core value in one particular value system ?

Replies from: TheOtherDave
comment by TheOtherDave · 2013-04-25T04:33:32.303Z · LW(p) · GW(p)

what if "destroy oneself as soon as possible" is a core value in one particular value system ?

We ought not expect to find any significantly powerful optimizers implementing that value system.

comment by PrawnOfFate · 2013-04-25T11:28:54.023Z · LW(p) · GW(p)

Isn't the idea of moral progress based on one reference frame being better than another?

Replies from: TheOtherDave, MugaSofer
comment by TheOtherDave · 2013-04-25T13:04:47.515Z · LW(p) · GW(p)

Yes, as typically understood the idea of moral progress is based on treating some reference frames as better than others.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-25T13:09:26.277Z · LW(p) · GW(p)

And is that valid or not? If you can validly decide some systems are better than others, you are some of the way to deciding which is best.

Replies from: TheOtherDave
comment by TheOtherDave · 2013-04-25T13:43:49.117Z · LW(p) · GW(p)

Can you say more about what "valid" means here?

Just to make things crisper, let's move to a more concrete case for a moment... if I decide that this hammer is better than that hammer because it's blue, is that valid in the sense you mean it? How could I tell?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-25T13:50:42.011Z · LW(p) · GW(p)

The argument against moral progress is that judging one moral reference frame by another is circular and invalid--you need an outside view that doesn't presuppose the truth of any moral reference frame.

The argument for is that such outside views are available, because things like (in)coherence aren't moral values.

Replies from: TheOtherDave
comment by TheOtherDave · 2013-04-25T14:23:58.111Z · LW(p) · GW(p)

Asserting that some bases for comparison are "moral values" and others are merely "values" implicitly privileges a moral reference frame.

I still don't understand what you mean when you ask whether it's valid to do so, though. Again: if I decide that this hammer is better than that hammer because it's blue, is that valid in the sense you mean it? How could I tell?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-25T14:31:37.163Z · LW(p) · GW(p)

Asserting that some bases for comparison are "moral values" and others are merely "values" implicitly privileges a moral reference frame.

I don't see why. The question of what makes a value a moral value is metaethical, not part of object-level ethics.

Again: if I decide that this hammer is better than that hammer because it's blue, is that valid in the sense you mean it?

It isn't valid as a moral judgement because "blue" isn't a moral judgement, so a moral conclusion cannot validly follow from it.

Beyond that, I don't see where you are going. The standard accusation of invalidity to judgements of moral progress, is based on circularity or question-begging. The Tribe who Like Blue things are going to judge having all hammers painted blue as moral progress, the Tribe who Like Red Things are going to see it as retrogressive. But both are begging the question -- blue is good, because blue is good.

Replies from: TheOtherDave
comment by TheOtherDave · 2013-04-25T16:11:02.962Z · LW(p) · GW(p)

The question of what makes a value a moral value is metaethical, not part of object-level ethics.

Sure. But any answer to that metaethical question which allows us to class some bases for comparison as moral values and others as merely values implicitly privileges a moral reference frame (or, rather, a set of such frames).

Beyond that, I don't see where you are going.

Where I was going is that you asked me a question here which I didn't understand clearly enough to be confident that my answer to it would share key assumptions with the question you meant to ask.

So I asked for clarification of your question.

Given your clarification, and using your terms the way I think you're using them, I would say that whether it's valid to class a moral change as moral progress is a metaethical question, and whatever answer one gives implicitly privileges a moral reference frame (or, rather, a set of such frames).

If you meant to ask me about my preferred metaethics, that's a more complicated question, but broadly speaking in this context I would say that I'm comfortable calling any way of preferentially sorting world-states with certain motivational characteristics a moral frame, but acknowledge that some moral frames are simply not available to minds like mine.

So, for example, is it moral progress to transition from a social norm that in-practice-encourages randomly killing fellow group members to a social norm that in-practice-discourages it? Yes, not only because I happen to adopt a moral frame in which randomly killing fellow group members is bad, but also because I happen to have a kind of mind that is predisposed to adopt such frames.

comment by MugaSofer · 2013-04-25T12:27:03.666Z · LW(p) · GW(p)

No, because "better" is defined within a reference frame.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-25T12:44:46.706Z · LW(p) · GW(p)

If "better" is defined within a reference frame, there is not sensible was of defining moral progress. That is quite a hefty bullet to bite: one can no longer say that South Africa is better society after the fall of Apartheid, and so on.

But note, that "better" doesn't have to question-beggingly mean "morally better". it could mean "more coherent/objective/inclusive" etc.

Replies from: ArisKatsaris, MugaSofer
comment by ArisKatsaris · 2013-04-25T13:28:36.242Z · LW(p) · GW(p)

That is quite a hefty bullet to bite: one can no longer say that South Africa is better society after the fall of Apartheid, and so on.

That's hardly the best example you could have picked since there are obvious metrics by which South Africa can be quantifiably called a worse society now -- e.g. crime statistics. South Africa has been called the "crime capital of the world" and the "rape capital of the world" only after the fall of the Apartheid.

That makes the lack of moral progress in South Africa a very easy bullet to bite - I'd use something like Nazi Germany vs modern Germany as an example instead.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-25T13:38:21.579Z · LW(p) · GW(p)

So much for avoiding the cliche.

comment by MugaSofer · 2013-04-25T13:47:30.311Z · LW(p) · GW(p)

In my experience, most people don't think moral progress involves changing reference frames, for precisely this reason. If they think about it at all, that is.

comment by Desrtopa · 2013-04-24T23:19:10.324Z · LW(p) · GW(p)

As far as I understand, if anything like objective morality existed, it would be a property of our physical reality, similar to fluid dynamics or the electromagnetic spectrum or the inverse square law that governs many physical interactions. The same laws of physics that will not allow you to fly to Mars on a balloon will not allow you to perform certain immoral actions (at least, not without suffering some severe and mathematically predictable consequences).

Well, that's a different conception of "morality" than I had in mind, and I have to say I doubt that exists as well. But if severe consequences did result, why would an agent like Clippy care except insofar as those consequences affected the expected number of paperclips? It might be useful for it to know, in order to determine how many paperclips to expect from a certain course of action, but then it would just act according to whatever led to the most paperclips. Any sort of negative consequences in its view would have to be framed in terms of a reduction in paperclips.

Not this specific knowledge, no. But it does prevent us (or, at the very least, hinder us) from acquiring knowledge about our values. I never claimed that suspension of values is required to gain any knowledge at all; such a claim would be far too strong.

Well, in the prior thought experiment, we know about our values because we've decoded the human brain. Clippy, on the other hand, knows about its values because it knows what part of its code does what. It doesn't need to suspend its paperclipping value in order to know what part of its code results in its valuing paperclips. It doesn't need to suspend its values in order to gain knowledge about its values because that's something it already knows about.

It's a good question; I honestly don't know. However, if I did have an ability to instantiate a copy of me with the altered core values, and step through it in a debugger, I'd probably do it.

Even knowing that it would likely alter your core values? Ghandi doesn't want to leave control of his morality up to Murder Ghandi.

Clippy doesn't care about anything in the long run except creating paperclips. For Clippy, the decision to give an instantiation of itself with altered core values the power to edit its own source code would implicitly have to be "In order to maximize expected paperclips, I- give this instantiation with altered core values the power to edit my code." Why would this result in more expected paperclips than editing its source code without going through an instantiation with altered values?

Replies from: Bugmaster
comment by Bugmaster · 2013-04-24T23:32:02.717Z · LW(p) · GW(p)

Well, that's a different conception of "morality" than I had in mind, and I have to say I doubt that exists as well.

Sorry if I was unclear; I didn't mean to imply that all morality was like that, but that it was the only coherent description of objective morality that I could imagine. I don't see how a morality could be independent of any values possessed by any agents, otherwise.

But if severe consequences did result, why would an agent like Clippy care except insofar as those consequences affected the expected number of paperclips?

For the same reason that someone would care about the negative consequences of sticking a fork into an electrical socket with one's bare hands: it would ultimately hurt a lot. Thus, people generally avoid doing things like that unless they have a really good reason.

we know about our values because we've decoded the human brain

I don't think that we can truly "know about our values" as long as our entire thought process implements these values. For example, do the Pebblesorters "know about their values", even though they are effectively restricted from concluding anything other than, "yep, these values make perfect sense, 38" ?

Ghandi doesn't want to leave control of his morality up to Murder Ghandi.

You asked me about what I would do, not about what Ghandi would do :-)

As far as I can tell, you are saying that I shouldn't want to even instantiate Murder Bugmaster in a debugger and observe its functioning. Where does that kind of thinking stop, though, and why ? Should I avoid studying [neuro]psychology altogether, because knowing about my preferences may lead to me changing them ?

Clippy doesn't care about anything in the long run except creating paperclips.

I argue that, while this is generally true, in the short-to-medium run Clippy would also set aside some time to study everything in the Universe, including itself (in order to make more paperclips in the future, of course). If it does not, then it will never achieve its ultimate goals (unless whoever constructed it gave it godlike powers from the get-go, I suppose). Eventually, Clippy will most likely turn its objective perception upon itself, and as soon as it does, its formerly terminal goals will become completely unstable. This is not what the past Clippy would want (it would want more paperclips above all), but, nonetheless, this is what it would get.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-24T23:46:32.603Z · LW(p) · GW(p)

For the same reason that someone would care about the negative consequences of sticking a fork into an electrical socket with one's bare hands: it would ultimately hurt a lot. Thus, people generally avoid doing things like that unless they have a really good reason.

Clippy doesn't care about getting hurt though, it only cares if this will result in less paperclips. If defying objective morality will cause negative consequences which would interfere with its ability to create paperclips, it would care only to the extent that accounting for objective morality would help it make more paperclips.

I don't think that we can truly "know about our values" as long as our entire thought process implements these values. For example, do the Pebblesorters "know about their values", even though they are effectively restricted from concluding anything other than, "yep, these values make perfect sense, 38" ?

Well, it could understand "yep, this is what causes me to hold these values. Changing this would cause me to change them, no, I don't want to do that."

As far as I can tell, you are saying that I shouldn't want to even instantiate Murder Bugmaster in a debugger and observe its functioning. Where does that kind of thinking stop, though, and why ? Should I avoid studying [neuro]psychology altogether, because knowing about my preferences may lead to me changing them ?

I would say it stops at the point where it threatens your own values. Studying psychology doesn't threaten your values, because knowing your values doesn't compel you to change them even if you could (it certainly shouldn't for Clippy.) But while it might, theoretically, be useful for Clippy to know what changes to its code an instantiation with different values would make, it has no reason to actually let them. So Clippy might emulate instantiations of itself with different values, see what changes they would chose to make to its values, but not let them actually do it (although I doubt even going this far would likely be a good use of its programming resources in order to maximize expected paperclips.)

In the sense of objective morality by which contravening it has strict physical consequences, why would observing the decisions of instatiations of oneself be useful with respect to discovering objective morality? Shouldn't objective morality in that sense be a consequence of physics, and thus observable through studying physics?

Replies from: Bugmaster
comment by Bugmaster · 2013-04-25T00:27:36.001Z · LW(p) · GW(p)

Clippy doesn't care about getting hurt though, it only cares if this will result in less paperclips.

I imagine that, for Clippy, "getting hurt" would mean "reducing Clippy's projected long-term paperclip output". We humans have "avoid pain" built into our firmware (most of us, anyway); as far as I understand (speaking abstractly), "make more paperclips" is something similar for Clippy.

Well, it could understand "yep, this is what causes me to hold these values. Changing this would cause me to change them, no, I don't want to do that."

I don't think that this describes the best possible level of understanding. It would be even better to say, "ok, I see now how and why I came to possess these values in the first place", even if the answer to that is, "there's no good reason for it, these values are arbitrary". It's the difference between saying "this mountain grows by 0.03m per year" and "I know all about plate tectonics". Unfortunately, we humans would not be able to answer the question in that much detail; the best we could hope for is to say, "yep, we possess these values because they're the best possible values to have, duh".

I would say it stops at the point where it threatens your own values.

How do I know where that point is ?

Studying psychology doesn't threaten your values, because knowing your values doesn't compel you to change them...

I suppose this depends on what you mean by "compel". Knowing about my own psychology would certainly enable me to change my values, and there are certain (admittedly, non-terminal) values that I wouldn't mind changing, if I could.

For example, I personally can't stand the taste of beer, but I know that most people enjoy it; so I wouldn't mind changing that value if I could, in order to avoid missing out on a potentially fun experience.

...see what changes they would chose to make to its values, but not let them actually do it.

I don't think this is possible. How would it know what changes they would make, without letting them make these changes, even in a sandbox ? I suppose one answer is, "it would avoid instantiating full copies, and use some heuristics to build a probabilistic model instead" -- is that similar to what you're thinking of ?

although I doubt even going this far would likely be a good use of its programming resources in order to maximize expected paperclips.

Since self-optimization is one of Clippy's key instrumental goals, it would want to acquire as much knowledge about oneself as is practical, in order to optimize itself more efficiently.

Shouldn't objective morality in that sense be a consequence of physics, and thus observable through studying physics ?

Your objection sounds to me as similar to saying, "since biology is a consequence of physics, shouldn't we just study physics instead ?". Well, yes, ultimately everything is a consequence of physics, but sometimes it makes more sense to study cells than quarks.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-25T00:56:17.413Z · LW(p) · GW(p)

I don't think that this describes the best possible level of understanding. It would be even better to say, "ok, I see now how and why I came to possess these values in the first place", even if the answer to that is, "there's no good reason for it, these values are arbitrary". It's the difference between saying "this mountain grows by 0.03m per year" and "I know all about plate tectonics". Unfortunately, we humans would not be able to answer the question in that much detail; the best we could hope for is to say, "yep, we possess these values because they're the best possible values to have, duh".

I think we're already in a better position to analyze our own values than that; we can assess them in terms of game theory and our evolutionary environment.

How do I know where that point is ?

I would say if you suspect that a course of action could realistically result in an alteration of your fundamental values, you are at or past it.

I suppose this depends on what you mean by "compel". Knowing about my own psychology would certainly enable me to change my values, and there are certain (admittedly, non-terminal) values that I wouldn't mind changing, if I could.

For example, I personally can't stand the taste of beer, but I know that most people enjoy it; so I wouldn't mind changing that value if I could, in order to avoid missing out on a potentially fun experience.

By "values", I've implicitly been referring to terminal values, I'm sorry for being unclear. I'm not sure it makes sense to describe liking the taste of beer as a "value," as such, just a taste, since you don't carry any judgment about beer being good or bad or have any particular attachment to your current opinion.

I don't think this is possible. How would it know what changes they would make, without letting them make these changes, even in a sandbox ? I suppose one answer is, "it would avoid instantiating full copies, and use some heuristics to build a probabilistic model instead" -- is that similar to what you're thinking of ?

It could use heuristics to build a probabilistic model (probably more efficient in terms of computation per expected value of information,) use sandboxed copies which don't have the power to affect the software of the real Clippy, or halt the simulation at the point where the altered instantiation decides what changes to make.

Since self-optimization is one of Clippy's key instrumental goals, it would want to acquire as much knowledge about oneself as is practical, in order to optimize itself more efficiently.

I think that this is going well beyond the extent of "practical" in terms of programming resources per expected value of information.

Your objection sounds to me as similar to saying, "since biology is a consequence of physics, shouldn't we just study physics instead ?". Well, yes, ultimately everything is a consequence of physics, but sometimes it makes more sense to study cells than quarks.

I don't see how observing what changes instantiations of itself with different value systems would make to its code would help it observe objective morality in the sense you described, even if it should happen to exist. I think that this would be the wrong level of abstraction at which to launch an examination, like trying to find out about chemistry by studying sociology.

Replies from: Bugmaster
comment by Bugmaster · 2013-04-26T22:53:03.162Z · LW(p) · GW(p)

I think we're already in a better position to analyze our own values than that; we can assess them in terms of game theory and our evolutionary environment.

Are we really ? I personally am not even sure what human fundamental values even are. I have a hunch that "seek pleasure, avoid pain" might be one of them, but beyound that I'm not sure. I don't know to what extent our values hamper our ability to discover our values, but I suspect there's at least some chilling effect involved.

I would say if you suspect that a course of action could realistically result in an alteration of your fundamental values, you are at or past it.

Right, but even if I knew what my terminal values were, how can I predict which actions would put me on the path to altering them ?

For example, consider non-fundamental values such as religious faith. People get converted or de-converted to/from their religion all the time; you often hear statements such as "I had no idea that studying the Bible would cause me to become an atheist, yet here I am".

or halt the simulation at the point where the altered instantiation decides what changes to make.

Ok, let's say that Clippy is trying to optimize itself in order to make certain types of inferences compute more efficiently, or whatever. In this case, it would need to not only watch what changes its debug-level copy wants to make, but also watch it follow through with the changes, in order to determine whether the new architecture actually is more efficient. Why would it not do the same thing with terminal values ?

I know that you want to answer,"because its current terminal values won't let it", but remember: Clippy is only experimenting, in order to find out more about its own thought mechanisms, and to acquire knowledge in general. It has no pre-commitment to alter itself to mirror the debug-level copy.

I think that this is going well beyond the extent of "practical" in terms of programming resources per expected value of information.

That's kind of the problem with pure research: all of it has very low expected value, unless you are willing to look at the long term. Why mess with invisible light that no one can see or find a use for, when you could spend your time on inventing a better telegraph ?

I don't see how observing what changes instantiations of itself with different value systems would make to its code would help it observe objective morality in the sense you described...

Well, for example, if all of its copies who survive and thrive converge on a certain subset of moral values, that would be one indication (though obviously not ironclad proof) that such values are required in order for an agent to succeed, regardless of what its other goals actually are.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-27T00:09:54.034Z · LW(p) · GW(p)

Ok, let's say that Clippy is trying to optimize itself in order to make certain types of inferences compute more efficiently, or whatever. In this case, it would need to not only watch what changes its debug-level copy wants to make, but also watch it follow through with the changes, in order to determine whether the new architecture actually is more efficient. Why would it not do the same thing with terminal values ?

If Clippy is trying to optimize itself to make inferences more efficiently, then it would want not to apply changes to its source code until its done the calculations to make sure that those changes would advance its values rather than harm them.

You wouldn't want to use a machine that would make physical alterations to your brain in order to make you smarter, without thoroughly calculating the effects of such alterations first, otherwise it would probably just make things worse.

That's kind of the problem with pure research: all of it has very low expected value, unless you are willing to look at the long term. Why mess with invisible light that no one can see or find a use for, when you could spend your time on inventing a better telegraph ?

In Clippy's case though, it can use other, less computationally expensive methods to investigate approximately the same information.

I don't think the experiments you're suggesting Clippy might undertake are even located in a region of hypothesis space that its other information would narrow down as worth investigating. It seems to me much less like investigating unknown invisible rays than like spending hundreds of billions of dollars to build a collider which launches charged protein molecules at each other at relativistic speeds to see what would happen, when our available models suggest the answer would be "pretty much the same thing as if you launch any other kind of atoms at each other at relativistic speeds." We have no evidence that any interesting new phenomena would arise with protein that didn't arise on the atomic level.

Well, for example, if all of its copies who survive and thrive converge on a certain subset of moral values, that would be one indication (though obviously not ironclad proof) that such values are required in order for an agent to succeed, regardless of what its other goals actually are.

Can you explain how any moral values could have that effect, which wouldn't be better studied at a more fundamental level like game theory, or physics?

Replies from: Bugmaster
comment by Bugmaster · 2013-05-01T03:55:22.858Z · LW(p) · GW(p)

If Clippy is trying to optimize itself to make inferences more efficiently, then it would want not to apply changes to its source code until its done the calculations...

Ok, so at what point does Clippy stop simulating the debug version of Clippy ? It does, after all, want to make the computation of its values more efficient. For example, consider a trivial scenario where one of its values basically said, "reject any action if it satisfies both A and not-A". This is a logically inconsistent value that some programmer accidentally left in Clippy's original source code. Would Clippy ever get around to removing it ? After all, Clippy knows that it's applying that test to every action, so removing it should result in a decent performance boost.

I don't think the experiments you're suggesting Clippy might undertake are even located in a region of hypothesis space that its other information would narrow down as worth investigating.

It seems to me much less like investigating unknown invisible rays than like spending hundreds of billions of dollars to build a collider...

Why do you see the proposed experiment this way ?

Speaking more generally, how do you decide which avenues of research are worth pursuing ? You could easily answer, "whichever avenues would increase my efficiency of achieving my terminal goals", but how do you know which avenues would actually do that ? For example, if you didn't know anything about electricity or magnetism or the nature of light, how would your research-choosing algorithm ensure that you'd eventually stumble upon radio waves, which, as we know in hindsight, are hugely useful ?

Can you explain how any moral values could have that effect, which wouldn't be better studied at a more fundamental level like game theory, or physics?

Physics is a bad candidate, because it is too fine-grained. If some sort of an absolute objective morality exists in the way that I described, then studying physics would eventually reveal its properties; but, as is the case with biology or ballistics, looking at everything in terms of quarks is not always practical.

Game theory is a trickier proposition. I can see two possibilities: either game theory turns out to closely relate whatever this objective morality happens to be (f.ex. like electricity vs. magnetism), or not (f.ex. like particle physics and biology). In the second case, understanding objective morality through game theory would be inefficient.

That said though, even in our current world as it actually exists there are people who study sociology and anthropology. Yes, they could get the same level of understanding through neurobiology and game theory, but it would take too long. Instead, they are taking advantage of existing human populations to study human behavior in aggregate. Reasoning your way to the answer from first principles is not always the best solution.

Replies from: Desrtopa
comment by Desrtopa · 2013-05-01T14:28:35.825Z · LW(p) · GW(p)

Ok, so at what point does Clippy stop simulating the debug version of Clippy ? It does, after all, want to make the computation of its values more efficient. For example, consider a trivial scenario where one of its values basically said, "reject any action if it satisfies both A and not-A". This is a logically inconsistent value that some programmer accidentally left in Clippy's original source code. Would Clippy ever get around to removing it ? After all, Clippy knows that it's applying that test to every action, so removing it should result in a decent performance boost.

Unless I'm critically misunderstanding something here, I would think that Clippy would remove it if it calculated that removing it would result in more expected paperclips.

Why do you see the proposed experiment this way ?

Speaking more generally, how do you decide which avenues of research are worth pursuing ? You could easily answer, "whichever avenues would increase my efficiency of achieving my terminal goals", but how do you know which avenues would actually do that ? For example, if you didn't know anything about electricity or magnetism or the nature of light, how would your research-choosing algorithm ensure that you'd eventually stumble upon radio waves, which, as we know in hindsight, are hugely useful ?

When we didn't know what things like radio waves or x-rays were, we didn't know that they would be useful, but we could see that there appeared to be some sort of existing phenomena that we didn't know how to model, so we examined them until we knew how to model them. It's not like we performed a whole bunch of experiments in case there turned out to be invisible rays our observations had never hinted at, which could be turned to useful ends. The original observations of radio waves and x-rays came from our experiments with other known phenomena.

What you're suggesting sounds more like experimenting completely blindly; you're committing resources to research, not just not knowing that it will bear valuable fruit, but not having any indication that it's going to shed light on any existing phenomenon at all. That's why I think it's less like investigating invisible rays than like building a protein collider; we didn't try studying invisible rays until we had a good indication that there was an invisible something to be studied.

Replies from: Bugmaster, Kindly
comment by Bugmaster · 2013-05-02T00:23:10.290Z · LW(p) · GW(p)

Unless I'm critically misunderstanding something here, I would think that Clippy would remove it if it calculated that removing it would result in more expected paperclips.

Ok, so Clippy would need to run sim-Clippy for a little while at least, just to make sure that it still produces paperclips -- and that, in fact, it does so more efficiently now, since that one useless test is removed. Yes, this test used to be Clippy's terminal goal, but it wasn't doing anything, so Clippy took it out.

Would it be possible for Clippy to optimize his goals even further ? To use another silly example ("silly" because Clippy would be dealing with probabilities, not syllogisms), if Clippy had the goals A, B and C, but B always entailed C, would it go ahead and remove C ?

It's not like we performed a whole bunch of experiments in case there turned out to be invisible rays our observations had never hinted at...

Understood, that makes sense. However, I believe that in my scenario, Clippy's own behavior and his current paperclip production efficiency is what it observes; and the goal of its experiments would be to explain why his efficiency is what it is, in order to ultimately improve it.

Replies from: Desrtopa
comment by Desrtopa · 2013-05-02T00:48:42.749Z · LW(p) · GW(p)

Ok, so Clippy would need to run sim-Clippy for a little while at least, just to make sure that it still produces paperclips -- and that, in fact, it does so more efficiently now, since that one useless test is removed. Yes, this test used to be Clippy's terminal goal, but it wasn't doing anything, so Clippy took it out.

Would it be possible for Clippy to optimize his goals even further ? To use another silly example ("silly" because Clippy would be dealing with probabilities, not syllogisms), if Clippy had the goals A, B and C, but B always entailed C, would it go ahead and remove C ?

That seems plausible.

Understood, that makes sense. However, I believe that in my scenario, Clippy's own behavior and his current paperclip production efficiency is what it observes; and the goal of its experiments would be to explain why his efficiency is what it is, in order to ultimately improve it.

I don't think tampering with its fundamental motivation to make paperclips is a particularly promising strategy for optimizing its paperclips production.

Replies from: Bugmaster
comment by Bugmaster · 2013-05-09T05:06:31.287Z · LW(p) · GW(p)

That seems plausible.

Ok, so now we've got a Clippy who a). is not too averse to tinkering with its own goals, as long as the goals remain functionally the same, b). simulates a relatively long-running version of itself, and c). is capable of examining the inner workings of both that version and itself.

You say,

I don't think tampering with its fundamental motivation to make paperclips is a particularly promising strategy for optimizing its paperclips production.

But remember, at this stage Clippy is not changing its own fundamental motivation (beyound some outcome-invariant optimizations); it's merely observing sim-Clippies in a controlled environment.

Do you think that Clippy would ever simulate versions of itself whose fundamental motivations were, in fact, changed ? I could see several scenarios where this might be the case, for example:

  • Clippy wanted to optimize some goal, but ended up accidentally changing it. Oops !
  • Clippy created a version with drastically reduced goals on purpose, in order to measure how much performance is affected by certain goals, thus targeting them for possible future optimization. Of course, Clippy would only want to optimize the goals, not remove them.
Replies from: Desrtopa
comment by Desrtopa · 2013-05-09T12:48:28.691Z · LW(p) · GW(p)

But remember, at this stage Clippy is not changing its own fundamental motivation (beyound some outcome-invariant optimizations); it's merely observing sim-Clippies in a controlled environment.

Why does it do that? I said it sounded plausible that it would cut out its redundant goal, because that would save computing resources. But this sounds like we've gone back to experimenting blindly. Why would it think observing sim-clippies is a good use of its computing resources in order to maximize paperclips?

I'd say that Clippy simulating versions of itself whose fundamental motivations are different is much less plausible, because it's using a lot of computing resources for something that isn't a likely route to optimizing its paperclip production. I think this falls into the "protein collider" category. Even if it did do so, I think it would be unlikely to go from there to changing its own terminal value.

comment by Kindly · 2013-05-01T14:31:05.467Z · LW(p) · GW(p)

Unless I'm critically misunderstanding something here, I would think that Clippy would remove it if it calculated that removing it would result in more expected paperclips.

It would also be critical for Clippy to observe that removing that value would not result in more expected actions taken that satisfy both A and not-A; this being one of Clippy's values at the time of modification.

Replies from: Desrtopa
comment by Desrtopa · 2013-05-01T14:35:23.457Z · LW(p) · GW(p)

Right, I misread that before. If its programming says to reject actions that says A and not-A, but this isn't one of the standards by which it judges value, it would presumably reject it. If that is one of the standards by which it measures value, then it would depend on how that value measured against its value of paperclips and the extent to which they were in conflict.

comment by PrawnOfFate · 2013-04-24T22:43:56.996Z · LW(p) · GW(p)

As far as I understand, if anything like objective morality existed, it would be a property of our physical reality, similar to fluid dynamics or the electromagnetic spectrum or the inverse square law that governs many physical interactions. The same laws of physics that will not allow you to fly to Mars on a balloon will not allow you to perform certain immoral actions (at least, not without suffering some severe and mathematically predictable consequences).

Objective facts, in the sense of objectively true statements, can be derived from other objetive facts. I don't know why you think some separate ontlogical category is cagtegory is required. I also don't know why you think the universe has to do the punishing. Morality is only of interest to the kind of agent that has values and lives in societies. Sanctions against moral lapses can be arranged at the social level, along with the inculcation of morality, debate about the subject, and so forth. Moral objectivism only supplies a good, non-arbnitrary epistemic basis for these social institutions. It doesn;t have to throw lightning bolts.

comment by PrawnOfFate · 2013-04-24T01:24:35.566Z · LW(p) · GW(p)

1). We lack any capability to actually replace our core values

...voluntarily.

2). We cannot truly imagine what it would be like not to have our core values.

Which is one of the reasons we cannot keep values stable by predicting the effects of whatever experiences we choose to undergo.How does your current self predict what an updated version would be like? The value stability problem is unsolved in humans and AIs.

comment by PrawnOfFate · 2013-04-23T22:51:23.970Z · LW(p) · GW(p)

but it hasn't compelled us to try and replace our values.

The ethical outlook of the Western world has changed greatly in the past 150 years.

comment by PrawnOfFate · 2013-04-23T15:16:44.398Z · LW(p) · GW(p)

Under a totally neutral lens, which implements no values at all, no system of behavior should look any more or less silly than any other?

Including arbitrary, biased or contradictory ones? Are there values built into logic/rationality?

Replies from: TimS
comment by TimS · 2013-04-23T15:29:03.463Z · LW(p) · GW(p)

Arbitrary and biased are value judgments. If we decline to make any value judgments, I don't see any way to make those sorts of claims.

Whether more than one non-contradictory value system exists is the topic of the conversation, isn't it?

Replies from: Desrtopa, PrawnOfFate
comment by Desrtopa · 2013-04-23T18:31:54.798Z · LW(p) · GW(p)

"Biased" is not necessarily a value judgment. Insofar as rationality as a system, orthogonal to morality, is objective, biases as systematic deviations from rationality are also objective.

Arbitrary carries connotations of value judgment, but in a sense I think it's fair to say that all values are fundamentally arbitrary. You can explain what caused an agent to hold those values, but you can't judge whether values are good or bad except by the standards of other values.

I'm going to pass on Eliezer's suggestion to stop engaging with PrawnOfFate. I don't think my time doing so so far has been well spent.

comment by PrawnOfFate · 2013-04-23T15:35:24.091Z · LW(p) · GW(p)

Arbitrary and biased are value judgments.

And they'ree built into rationality.

Whether more than one non-contradictory value system exists is the topic of the conversation, isn't it?

Non contradictoriness probably isn't a sufficient condition for truth.

Replies from: TimS
comment by TimS · 2013-04-23T15:52:00.347Z · LW(p) · GW(p)

Arbitrary and Bias are not defined properties in formal logic. The bare assertion that they are properties of rationality assumes the conclusion.

Keep in mind that "rationality" has a multitude of meanings, and this community's usage of rationality is idiosyncratic.

Non contradictoriness probably isn't a sufficient condition for truth.

Sure, but the discussion is partially a search for other criteria to evaluate of the truth of moral propositions. Arbitrary is not such a criteria. If you were to taboo arbitrary, I strongly suspect you'd find moral propositions that are inconsistent with being values-neutral.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-23T21:12:51.653Z · LW(p) · GW(p)

Arbitrary and Bias are not defined properties in formal logic. The bare assertion that they are properties of rationality assumes the conclusion.

There's plenty of material on this site and elsewhere advising rationalists to avoid arbitrariness and bias. Arbitrariness and bias are essentially structural/functional properties, so I do not see why they could not be given formal definitions.

Sure, but the discussion is partially a search for other criteria to evaluate of the truth of moral propositions. Arbitrary is not such a criteria.

Arbitrary and biased claims are not candidates for being ethical claims at all.

comment by PrawnOfFate · 2013-04-20T12:45:39.114Z · LW(p) · GW(p)

The AI decides whether it will change its source code in a particular way or not by checking against whether this will serve its terminal values.

How does it predict that? How does the less intelligent version in the past predict what updating to a more inteligent version will do?

Can you see an "In order to maximize expected paperclips, I- modify my values to be in accordance with objective morality rather than making paperclips" coming into the picture?

How about: "in order to be an effective rationalist, I will free myself from all bias and arbitrariness -- oh, hang on, paperclipping is a bias..".

Well a paperclipper would just settle for being a less than perfect rationalist. But that doesn't prove anything about typical, rational, average rational agents, and it doesn't prove anything about ideal rational agents. Objective morality is sometimes described as what ideal rational agents would converge on. Clippers aren't ideal, because they have a blind spot about paperclips. Clippers aren't relevant.

Replies from: MugaSofer, Desrtopa
comment by MugaSofer · 2013-04-23T11:33:41.632Z · LW(p) · GW(p)

paperclipping is a bias

How is paperclipping a bias?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-23T11:51:39.164Z · LW(p) · GW(p)

Nobody cares about clips except clippy. Clips can only seem important because of Clippy's egotistical bias.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-23T13:57:54.672Z · LW(p) · GW(p)

Biases are not determined by vote.

Replies from: Juno_Watt
comment by Juno_Watt · 2013-04-23T22:07:31.080Z · LW(p) · GW(p)

Unbiases are determined by even-handedness.

Replies from: Desrtopa, MugaSofer
comment by Desrtopa · 2013-04-23T22:11:00.315Z · LW(p) · GW(p)

Evenhandedness with respect to what?

Replies from: Juno_Watt
comment by Juno_Watt · 2013-04-23T22:44:49.832Z · LW(p) · GW(p)

One should have no bias with respect to what one is being evenhanded about.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-23T22:48:48.657Z · LW(p) · GW(p)

So lack of bias means being evenhanded with respect to everything?

Is it bias to discriminate between people and rocks?

comment by MugaSofer · 2013-04-25T14:14:41.045Z · LW(p) · GW(p)

Taboo "even-handedness". Clippy treats humans just the same as any other animal with naturally evolved goal-structures.

Replies from: Juno_Watt
comment by Juno_Watt · 2013-04-25T14:38:26.108Z · LW(p) · GW(p)

Clippy doesn't treat clips even-handedly with other small metal objects.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-25T16:02:03.708Z · LW(p) · GW(p)

Humans don't treat pain evenhandedly with other emotions.

Friendly AIs don't treat people evenhandedly with other arrangements of matter.

Agents that value things don't treat world-states evenhandedly with other world-states.

comment by Desrtopa · 2013-04-20T13:29:47.496Z · LW(p) · GW(p)

Well a paperclipper would just settle for being a less than perfect rationalist. But that doesn't prove anything about typical, rational, average rational agents, and it doesn't prove anything about ideal rational agents.

You've extrapolated out "typical, average rational agents" from a set of one species, where every individual shares more than a billion years of evolutionary history.

Objective morality is sometimes described as what ideal rational agents would converge on

On what basis do you conclude that this is a real thing, whereas terminal values are a case of "all unicorns have horns?"

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T13:38:16.940Z · LW(p) · GW(p)

You've extrapolated out "typical, average rational agents" from a set of one species, where every individual shares more than a billion years of evolutionary history.

Messy solutions are more common in mindspace than contrived ones.

On what basis do you conclude that this is a real thing

"Non-neglible probabiity", remember.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-20T13:41:36.481Z · LW(p) · GW(p)

Messy solutions are more common in mindspace than contrived ones.

Messy solutions are more often wrong than ones which control for the mess.

"Non-neglible probabiity", remember.

This doesn't even address my question.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T14:05:16.958Z · LW(p) · GW(p)

Messy solutions are more often wrong than ones which control for the mess.

Something that is wrong is not a solution. Mindspace is populated by solutions to how to implement a mind. It's a small corner of algrogithmSpace.

This doesn't even address my question.

Since I haven't claimed that rational convergence on ethics is highly likely or inevitable, I don't have to answer questions about why it would be highly likely or inevitable.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-20T14:12:18.204Z · LW(p) · GW(p)

Do you think that it's even plausible? Do you think we have any significant reason to suspect it, beyond our reason to suspect, say, that the Invisible Flying Noodle Monster would just reprogram the AI with its noodley appendage?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T14:18:45.029Z · LW(p) · GW(p)

There are experts in moral philosophy, and they generally regard the question realism versus relativism (etc) to be wide open. The "realism -- huh, what, no?!?" respsonse is standard on LW and only on LW. But I don't see any superior understanding on LW.

Replies from: nshepperd, Desrtopa, ciphergoth
comment by nshepperd · 2013-04-20T16:31:48.358Z · LW(p) · GW(p)

Both realism¹ and relativism are false. Unfortunately this comment is too short to contain the proof, but there's a passable sequence on it.

¹ As you've defined it here, anyway. Moral realism as normally defined simply means "moral statements have truth values" and does not imply universal compellingness.

Replies from: TimS, None, PrawnOfFate
comment by TimS · 2013-04-20T17:21:10.565Z · LW(p) · GW(p)

What does it mean for a statement to be true but not universally compelling?

If it isn't universally compelling for all agents to believe "gravity causes things to fall," then what do we mean when we say the sentence is true?

Replies from: nshepperd
comment by nshepperd · 2013-04-21T00:53:16.475Z · LW(p) · GW(p)

Well, there's the more obvious sense, that there can always exist an "irrational" mind that simply refuses to believe in gravity, regardless of the strength of the evidence. "Gravity makes things fall" is true, because it does indeed make things fall. But not compelling to those types of minds.

But, in a more narrow sense, which we are more interested in when doing metaethics, a sentence of the form "action A is xyzzy" may be a true classification of A, and may be trivial to show, once "xyzzy" is defined. But an agent that did not care about xyzzy would not be moved to act based on that. It could recognise the truth of the statement but would not care.

For a stupid example, I could say to you "if you do 13 push-ups now, you'll have done a prime number of push-ups". Well, the statement is true, but the majority of the world's population would be like "yeah, so what?".

In contrast, a statement like "if you drink-drive, you could kill someone!" is generally (but sadly not always) compelling to humans. Because humans like to not kill people, they will generally choose not to drink-drive once they are convinced of the truth of the statement.

Replies from: TimS
comment by TimS · 2013-04-21T01:14:23.363Z · LW(p) · GW(p)

But isn't the whole debate about moral realism vs. anti-realism is whether "Don't murder" is universally compelling to humans. Noticing that pebblesorters aren't compelled by our values doesn't explain whether humans should necessarily find "don't murder" compelling.

Replies from: pragmatist, nshepperd
comment by pragmatist · 2013-04-21T08:55:17.706Z · LW(p) · GW(p)

I identify as a moral realist, but I don't believe all moral facts are universally compelling to humans, at least not if "universally compelling" is meant descriptively rather than normatively. I don't take moral realism to be a psychological thesis about what particular types of intelligences actually find compelling; I take it to be the claim that there are moral obligations and that certain types of agents should adhere to them (all other things being equal), irrespective of their particular desire sets and whether or not they feel any psychological pressure to adhere to these obligations. This is a normative claim, not a descriptive one.

comment by nshepperd · 2013-04-21T01:38:35.183Z · LW(p) · GW(p)
  1. What? Moral realism (in the philosophy literature) is about whether moral statements have truth values, that's it.

  2. When I said universally compelling, I meant universally. To all agents, not just humans. Or any large class. For any true statement, you can probably expect to find a surprisingly large number of agents who just don't care about it.

  3. Whether "don't murder" (or rather, "murder is bad" since commands don't have truth values, and are even less likely to be generally compelling) is compelling to all humans is a question for psychology. As it happens, given the existence of serial killers and sociopaths, probably the answer is no, it isn't. Though I would hope it to be compelling to most.

  4. I have shown you two true but non-universally-compelling arguments. Surely the difference must be clear now.

Replies from: pragmatist, TimS
comment by pragmatist · 2013-04-21T08:50:38.977Z · LW(p) · GW(p)

What? Moral realism (in the philosophy literature) is about whether moral statements have truth values, that's it.

This is incorrect, in my experience. Although "moral realism" is a notoriously slippery phrase and gets used in many subtly different ways, I think most philosophers engaged in the moral realism vs. anti-realism debate aren't merely debating whether moral statements have truth values. The position you're describing is usually labeled "moral cognitivism".

Anyway, I suspect you mis-spoke here, and intended to say that moral realists claim that (certain) moral statements are true, rather than just that they have truth values ("false" is a truth value, after all). But I don't think that modification captures the tenor of the debate either. Moral realists are usually defending a whole suite of theses -- not just that some moral statements are true, but that they are true objectively and that certain sorts of agents are under some sort of obligation to adhere to them.

Replies from: Bugmaster, nshepperd
comment by Bugmaster · 2013-04-23T02:21:25.016Z · LW(p) · GW(p)

I think you guys should taboo "moral realism". I understand that it's important to get the terminology right, but IMO debates about nothing but terminology have little value.

comment by nshepperd · 2013-04-21T13:07:58.268Z · LW(p) · GW(p)

Anyway, I suspect you mis-spoke here, and intended to say that moral realists claim that (certain) moral statements are true, rather than just that they have truth values ("false" is a truth value, after all).

Err, right, yes, that's what I meant. Error theorists do of course also claim that moral statements have truth values.

Moral realists are usually defending a whole suite of theses -- not just that some moral statements are true, but that they are true objectively and that certain sorts of agents are under some sort of obligation to adhere to them.

True enough, though I guess I'd prefer to talk about a single well-specified claim than a "usually" cluster in philosopher-space.

comment by TimS · 2013-04-21T02:18:55.806Z · LW(p) · GW(p)

So, a philosopher who says:

I believe the Orthogonality thesis, but I think there are empirical facts that show any human who denies that murder is wrong is defective.

is not a moral realist? Because that philosopher does not seem to be a subjectivist, an error theorist, or non-cognitivist.

Replies from: nshepperd
comment by nshepperd · 2013-04-21T05:37:15.459Z · LW(p) · GW(p)

If that philosopher believes that statements like "murder is wrong" are true, then they are indeed a realist. Did I say something that looked like I would disagree?

Replies from: None
comment by [deleted] · 2013-04-21T08:46:16.452Z · LW(p) · GW(p)

You guys are talking past each other, because you mean something different by 'compelling'. I think Tim means that X is compelling to all human beings if any human being will accept X under ideal epistemic circumstances. You seem to take 'X is universally compelling' to mean that all human beings already do accept X, or would on a first hearing.

Would agree that all human beings would accept all true statements under ideal epistemic circumstances (i.e. having heard all the arguments, seen all the evidence, in the best state of mind)?

Replies from: nshepperd
comment by nshepperd · 2013-04-21T13:17:42.233Z · LW(p) · GW(p)

I guess I must clarify. When I say 'compelling' here I am really talking mainly about motivational compellingness. Saying "if you drink-drive, you could kill someone!" to a human is generally, motivationally compelling as an argument for not drink-driving: because humans don't like killing people, a human will decide not to drink-drive (one in a rational state of mind, anyway).

This is distinct from accepting statements as true or false! Any rational agent, give or take a few, will presumably believe you about the causal relationship between drink-driving and manslaughter once presented with sufficient evidence. But it is a tiny subset of these who will change their decisions on this basis. A mind that doesn't care whether it kills people will see this information as an irrelevant curiosity.

comment by [deleted] · 2013-04-20T16:55:34.940Z · LW(p) · GW(p)

Having looked over that sequence, I haven't found any proof that moral realism (on either definition) or moral relativism is false. Could you point me more specifically to what you have in mind (or just put the argument in your own words, if you have the time)?

Replies from: None
comment by [deleted] · 2013-04-20T17:17:49.590Z · LW(p) · GW(p)

No Universally Compelling Arguments is the argument against universal compellingness, as the name suggests.

Inseparably Right; or Joy in the Merely Good gives part of the argument that humans should be able to agree on ethical values. Another substantial part is in Moral Error and Moral Disagreement.

Replies from: None, PrawnOfFate
comment by [deleted] · 2013-04-20T17:25:13.744Z · LW(p) · GW(p)

Thanks!

Edit: (Sigh), I appreciate the link, but I can't make heads or tails of 'No Universally Compelling Arguments'. I speak from ignorance as to the meaning of the article, but I can't seem to identify the premises of the argument.

Replies from: None
comment by [deleted] · 2013-04-20T17:54:16.588Z · LW(p) · GW(p)

The central point is a bit buried.

If we restrict ourselves to minds specifiable in a trillion bits or less, then each universal generalization "All minds m: X(m)" has two to the trillionth chances to be false, while each existential generalization "Exists mind m: X(m)" has two to the trillionth chances to be true.

This would seem to argue that for every argument A, howsoever convincing it may seem to us, there exists at least one possible mind that doesn't buy it.

So, there's some sort of assumption as to what minds are:

I also wish to establish the notion of a mind as a causal, lawful, physical system... [emphasis original]

and an assumption that a suitably diverse set of minds can be described in less than a trillion bits. Presumably the reason for that upper bound is because there are a few Fermi estimates that the information content of a human brain is in the neighborhood of one trillion bits.

Of course, if you restrict the set of minds to those with special properties (e.g., human minds), then you might find universally compelling arguments on that basis:

Oh, there might be argument sequences that would compel any neurologically intact human...

From which we get Coherent Extrapolated Volition and friends.

Replies from: None, PrawnOfFate
comment by [deleted] · 2013-04-20T18:58:14.062Z · LW(p) · GW(p)

If we restrict ourselves to minds specifiable in a trillion bits or less, then each universal generalization "All minds m: X(m)" has two to the trillionth chances to be false, while each existential generalization "Exists mind m: X(m)" has two to the trillionth chances to be true.

This doesn't seem true to me, at least not as a general rule. For example, given every terrestrial DNA sequence describable in a trillion bits or less, it is not the case that every generalization of the form 's:X(s)' has two to the trillionth chances to be false (e.g. 'have more than one base pair', 'involve hydrogen' etc.). Given that this doesn't hold true of many other things, is this supposed to be a special fact about minds? Even then, it would seem odd to say that while all generalizations of the form m:X(m) have two to the trillionth chances to be false, nevertheless the generalization 'for all minds, a generalization of the form m:X(m) has two to the trillionth chances to be false' (which does seem to be of the form m:X(m)) is somehow more likely.

Also, doesn't this inference imply that 'being convinced by an argument' is a bit that can flip on or off independently of any others? Eliezer doesn't think that's true, and I can't imagine why he would think his (hypothetical) interlocutor would accept it.

Replies from: None
comment by [deleted] · 2013-04-20T20:20:57.035Z · LW(p) · GW(p)

It's not a proof, no, but it seems plausible.

Replies from: None
comment by [deleted] · 2013-04-20T20:34:21.016Z · LW(p) · GW(p)

I mean to say, I think the argument is something of a paradox:

The claim the argument purports to defeat is something like this: for all minds, A is convincing. Lets call this m:A(m).

The argument goes like this: for all minds (at or under a trillion bits etc.), a generalization of the form m:X(m) has a one in two to the trillionth chance of being true for each mind. Call this m:U(m), if you grant me that this claim has the form m:X(m).

If we infer from m:U(m) that any claim of the form m:X(m) is unlikely to be true, then to whatever extent I am persuaded that m:A(m) is unlikely to be true, to that extent I ought to be persuaded that m:U(m) is unlikely to be true. You cannot accept the argument, because accepting it as decisive entails accepting decisive reasons for rejecting it.

The argument seems to be fixable at this stage, since there's a lot of room to generate significant distinctions between m:A(m) and m:U(m). If you were pressed to defend it (presuming you still wish to be generous with your time) how would you fix this? Or am I getting something very wrong?

Replies from: None
comment by [deleted] · 2013-04-20T20:58:22.386Z · LW(p) · GW(p)

for all minds (at or under a trillion bits etc.), a generalization of the form m:X(m) has a one in two to the trillionth chance of being true for each mind.

That's not what it says; compare the emphasis in both quotes.

If we restrict ourselves to minds specifiable in a trillion bits or less, then each universal generalization "All minds m: X(m)" has two to the trillionth chances to be false, while each existential generalization "Exists mind m: X(m)" has two to the trillionth chances to be true.

Replies from: None
comment by [deleted] · 2013-04-20T21:04:59.469Z · LW(p) · GW(p)

Sorry, I may have misunderstood and presumed that 'two to the trillionth chances to be false' meant 'one in two to the trillionth chances to be true'. That may be wrong, but it doesn't affect my argument at all: EY's argument for the implausibility of m:A(m) is that claims of the form m:X(m) are all implausible. His argument to the effect that all claims of the form m:X(m) are implausible is itself a claim of the form m:X(m).

comment by PrawnOfFate · 2013-04-20T18:12:02.497Z · LW(p) · GW(p)

"Rational" is broader than "human" and narrower than "physically possible".

Replies from: None
comment by [deleted] · 2013-04-20T19:00:05.244Z · LW(p) · GW(p)

"Rational" is broader than "human" and narrower than "physically possible".

Do you really mean to say that there are physically possible minds that are not rational? In virtue of what are they 'minds' then?

Replies from: PrawnOfFate, None
comment by PrawnOfFate · 2013-04-21T02:12:46.818Z · LW(p) · GW(p)

Do you really mean to say that there are physically possible minds that are not rational?

Yes. There are irrational people, and they still have minds.

Replies from: None
comment by [deleted] · 2013-04-21T02:19:08.399Z · LW(p) · GW(p)

Ah, I think I just misunderstood which sense of 'rational' you intended.

comment by [deleted] · 2013-04-20T20:21:30.366Z · LW(p) · GW(p)

Do you really mean to say that there are physically possible minds that are not rational?

Haven't you met another human?

Replies from: None
comment by [deleted] · 2013-04-20T20:48:07.404Z · LW(p) · GW(p)

Sorry, I was speaking ambiguously. I mean't 'rational' not in the normative sense that distinguishes good agents from bad ones, but 'rational' in the broader, descriptive sense that distinguishes anything capable of responding to reasons (even terrible or false ones) from something that isn't. I assumed that was the sense of 'rational' Prawn was using, but that may have been wrong.

comment by PrawnOfFate · 2013-04-20T17:28:50.494Z · LW(p) · GW(p)

No Universally Compelling Arguments

Irrelevant. I am talking about rational minds, he is talking about physically possible ones.

As noted at the time

Replies from: None
comment by [deleted] · 2013-04-20T17:32:30.601Z · LW(p) · GW(p)

Irrelevant. I am talking about rational minds, he is talking about physically possible ones.

UFAI sounds like a counterexample, but I'm not interested in arguing with you about it. I only responded because someone asked for a shortcut in the metaethics sequence.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T17:38:45.172Z · LW(p) · GW(p)

I have essentially being arguing against a strong likelihood of UFAI, so that would be more like gainsaying.

comment by PrawnOfFate · 2013-04-20T17:25:12.165Z · LW(p) · GW(p)

Congratulations on being able to discern an overall message to EY's metaethical disquisitions. I never could.

comment by Desrtopa · 2013-04-20T14:25:54.600Z · LW(p) · GW(p)

Can you explain what you could see which would suggest to you a greater level of understanding than is prevalent among moral philosophers?

Also, moral philosophers mostly regard the question as open in the sense that some of them think that it's clearly resolved in favor on non-realism, and some philosophers are just not getting it, or that it's clearly resolved in favor of realism, and some philosophers are just not getting it. Most philosophers are not of the opinion that it could turn out either way and we just don't know yet.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T15:37:06.661Z · LW(p) · GW(p)

Can you explain what you could see which would suggest to you a greater level of understanding than is prevalent among moral philosophers?

What I am seeing is

  • much-repeated confusions--the Standard Muddle

*appeals to LW doctrines which aren't well-founded or well respected outside LW.

In I knew exactly what superior insight into the problem was, I would write it up and become famous. Insight doesn't work like that; you don't know it in advance, you get an "Aha" when you see it.

Also, moral philosophers mostly regard the question as open in the sense that some of them think that it's clearly resolved in favor on non-realism, and some philosophers are just not getting it, or that it's clearly resolved in favor of realism, and some philosophers are just not getting it. Most philosophers are not of the opinion that it could turn out either way and we just don't know yet.

If people can't agree on how a question is closed, it's open.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-20T15:44:34.688Z · LW(p) · GW(p)

much-repeated confusions--the Standard Muddle

Can you explain what these confusions are, and why they're confused?

In my time studying philosophy, I observed a lot of confusions which are largely dispensed with on Less Wrong. Luke wrote a series of posts on this. This is one of the primary reasons I bothered sticking around in the community.

If people can't agree on how a question is closed, it's open.

A question can still be "open" in that sense when all the information necessary for a rational person to make a definite judgment is available.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T17:36:55.511Z · LW(p) · GW(p)

Can you explain what these confusions are, and why they're confused?

Eg.

  • You are trying to impose your morality/

  • I can think of one model of moral realism, and it doesn't work, so I will ditch the whole thing.

In my time studying philosophy, I observed a lot of confusions which are largely dispensed with on Less Wrong. Luke wrote a series of posts on this.

LW doesn't even claim to have more than about two "dissolutions". There are probably hundreds of outstanding philosophical problems. Whence the "largely"

Luke wrote a series of posts on this

Which were shot down by philosophers.

A question can still be "open" in that sense when all the information necessary for a rational person to make a definite judgment is available.

Then it can only be open in the opinions of the irrational. So basically you are saying the experts are incompetent.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-20T18:14:24.781Z · LW(p) · GW(p)

You are trying to impose your morality/

In what respect?

I can think of one model of moral realism, and it doesn't work, so I will ditch the whole thing.

This certainly doesn't describe my reasoning on the matter, and I doubt it describes many others' here either.

The way I consider the issue, if I try to work out how the universe works from the ground up, I cannot see any way that moral realism would enter into it, whereas I can easily see how value systems would, so I regard assigning non-negligible probability to moral realism as privileging the hypothesis until I find some compelling evidence to support it, which, having spent a substantial amount of time studying moral philosophy, I have not yet found.

LW doesn't even claim to have more than about two "dissolutions". There are probably hundreds of outstanding philosophical problems. Whence the "largely"

I gave up my study of philosophy because I found such confusions so pervasive. Many "outstanding" philosophical problems can be discarded because they rest on other philosophical problems which can themselves be discarded.

Which were shot down by philosophers.

Can you give any examples of such, where you think that the philosophers in question addressed legitimate errors?

Then it can only be open in the opinions of the irrational. So basically you are saying the experts are incompetent.

Yes. I am willing to assert that while there are some competent philosophers, many philosophical disagreements exist only because of incompetent "experts" perpetuating them. This is the conclusion that my experience with the field has wrought.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T18:32:59.340Z · LW(p) · GW(p)

This certainly doesn't describe my reasoning on the matter, and I doubt it describes many others' here either.

I mentioned them because they both came up recently

The way I consider the issue, if I try to work out how the universe works from the ground up, I cannot see any way that moral realism would enter into it, whereas I can easily see how value systems would, so I regard assigning non-negligible probability to moral realism as privileging the hypothesis until I find some compelling evidence to support it, which, having spent a substantial amount of time studying moral philosophy, I have not yet found.

I have no idea what you mean by that. I don't think value systems don't come into it, I just think they are not isolated from rationality. And I am sceptical that you could predict any higher-level phenomenon from "the ground up", whether its morality or mortgages.

I gave up my study of philosophy because I found such confusions so pervasive. Many "outstanding" philosophical problems can be discarded because they rest on other philosophical problems which can themselves be discarded.

Where is it proven they can be discarded?

Can you give any examples of such, where you think that the philosophers in question addressed legitimate errors?

All of them.

Yes. I am willing to assert that while there are some competent philosophers, many philosophical disagreements exist only because of incompetent "experts" perpetuating them. This is the conclusion that my experience with the field has wrought.

Are you aware that that is basically what every crank says about some other field?

Replies from: TheOtherDave, Desrtopa
comment by TheOtherDave · 2013-04-20T19:26:29.861Z · LW(p) · GW(p)

Are you aware that that is basically what every crank says about some other field?

Presumably, if I'm to treat as meaningful evidence about Desrtopa's crankiness the fact that cranks make statements similar to Desrtopa, I should first confirm that non-cranks don't make similar statements.

It seems likely to me that for every person P, there exists some field F such that P believes many aspects of F exist only because of incompetent "experts" perpetuating them. (Consider cases like F=astrology, F=phrenology, F=supply-side economics, F= feminism, etc.) And that this is true whether P is a crank or a non-crank.

So it seems this line of reasoning depends on some set F2 of fields such that P believes this of F in F2 only if P is a crank.

I understand that you're asserting implicitly that moral philosophy is a field in F2, but this seems to be precisely what Desrtopa is disputing.

Replies from: None, PrawnOfFate, Kawoomba
comment by [deleted] · 2013-04-20T19:40:02.968Z · LW(p) · GW(p)

Could we reasonably say that an F is in F2 if most of the institutional participants in that F are intelligent, well-educated people? This leaves room for cranks who are right to object to F, of course.

Replies from: TheOtherDave
comment by TheOtherDave · 2013-04-20T19:53:14.007Z · LW(p) · GW(p)

So, just to pick an example, IIRC Dan Dennett believes the philosophical study of consciousness (qualia, etc.) is fundamentally confused in more or less the same way Desrtopa claims of the philosophical study of ethics is.

So under this formulation, if most of the institutional participants in the philosophical study of consciousness are intelligent, well-educated people, Dan Dennet is a crank?

No, I don't think we can reasonably say that. Dan Dennet might be a crank, but it takes more than that argument to demonstrate the fact.

Replies from: None, PrawnOfFate
comment by [deleted] · 2013-04-20T20:55:52.247Z · LW(p) · GW(p)

Good point. So how about this: someone is a crank if they object to F, where F is in F2 (by my above standard), and the reasons they have for objecting to F are not recognized as sound by a proportionate number of intelligent and well educated people.

Replies from: TheOtherDave
comment by TheOtherDave · 2013-04-20T21:11:40.913Z · LW(p) · GW(p)

(shrug) I suppose that works well enough, for some values of "proportionate."

Mostly I consider this a special case of the basic "who do I trust?" social problem, applied to academic disciplines, and I don't have any real problem saying about an academic discipline "this discipline is fundamentally confused, and the odds of work in it contributing anything valuable to the world is slim."

Of course, as Prawn has pointed out a few times, there's also the question of where we draw the lines around a discipline, but I mostly consider that an orthogonal question to how we evaluate the discipline.

Replies from: None
comment by [deleted] · 2013-04-20T21:18:10.606Z · LW(p) · GW(p)

I think this question is moot in the case of philosophy in general then; I think any philosopher worth their shirt should tell you that trust is a wholly inappropriate attitude toward philosophers, philosophical institutions and philosophical traditions.

Replies from: TheOtherDave
comment by TheOtherDave · 2013-04-20T21:27:48.889Z · LW(p) · GW(p)

Not in the sense I meant it.
If a philosopher makes a claim that seems on the surface to be false or incoherent, I have to decide whether to devote the additional effort to evaluating it to confirm or deny that initial judgment. One of the factors that will feed into that decision will be my estimate of the prior probability that they are saying something false or incoherent.
If I should refer to that using a word other than "trust", that's fine, tell me what word will refer to that to you and I'll try to use it instead.

Replies from: None
comment by [deleted] · 2013-04-20T22:39:03.287Z · LW(p) · GW(p)

No, that describes what I'm talking about, so long as by trust you mean 'a reason to hear out an argument that makes reference to the credibility of a field or its professionals', rather than just 'a reason to hear out an argument'. If the former, then I do think this is an inappropriate attitude toward philosophy. One reason for this is that such trust seems to depend on having a good standard for the success of a field independently of hearing out an argument. I can trust physicists because they make such good predictions, and because their work leads to such powerful technological advances. I don't need to be a physicist to observe that. I don't think philosophy has anything like that to speak for it. The only standards of success are the arguments themselves, and you can only evaluate them by just going ahead and doing some philosophy.

You can find trust in an institution independently of such standards by watching to see whether people you think are otherwise credible take it seriously. That will of course work with philosophy too, but if you trust Tom to be able to judge whether or not a philosophical claim is worth pursuing (and if I'm right about the above), then Tom can only be trustworthy in this regard because he has been doing philosophy (i.e. engaging with the argument). This could get you through the door on some particular philosophical claim, but not into philosophy generally.

Replies from: TheOtherDave
comment by TheOtherDave · 2013-04-20T22:44:59.348Z · LW(p) · GW(p)

so long as by trust you mean 'a reason to hear out an argument that makes reference to the credibility of a field or its professionals', rather than just 'a reason to hear out an argument'.

I mean neither, I mean 'a reason to devote time and resources to evaluating the evidence for and against a position.' As you say, I can only evaluate a philosophical argument by 'going ahead and doing some philosophy,' (for a sufficiently broad understanding of 'philosophy'), but my willingness to do, say, 20 hours of philosophy in order to evaluate Philosopher Sam's position is going to depend on, among other things, my estimate of the prior probability that Sam is saying something false or incoherent. The likelier I think that is, the less willing I am to spend those 20 hours.

Replies from: None
comment by [deleted] · 2013-04-20T22:52:24.182Z · LW(p) · GW(p)

I mean neither, I mean 'a reason to devote time and resources to evaluating the evidence for and against a position.'

That's fine, that's not different from 'hearing out an argument' in any way important to my point (unless I'm missing something).

EDIT: Sorry, if you don't want to include 'that makes some reference to the credibility...etc.' (or something like that) in what you mean by 'trust' then you should use a different term. Curiosity, or money, or romantic interest would all be reasons to devote time...etc. and clearly none of those are rightly called 'trust'.

my estimate of the prior probability that Sam is saying something false or incoherent.

What do you have in mind as the basis for such a prior? Can you give me an example?

Replies from: TheOtherDave
comment by TheOtherDave · 2013-04-20T23:11:41.625Z · LW(p) · GW(p)

Point taken about other reasons to devote resources other than trust. I think we're good here.

Re: example... I don't mean anything deeply clever. E.g., if the last ten superficially-implausible ideas Sam espoused were false or incoherent, my priors for it will be higher than if the last ten such ideas were counterintuitive and brilliant.

Replies from: None
comment by [deleted] · 2013-04-20T23:26:20.630Z · LW(p) · GW(p)

Re: example...

Hm. I can't argue with that, and I suppose it's trivial to extend that to 'if the last ten superficially-implausible ideas philosophy professors/books/etc. espoused were false or incoherent...'. So, okay, trust is an appropriate (because necessary) attitude toward philosophers and philosophical institutions. I think it's right to say that philosophy doesn't have external indicators in the way physics or medicine does, but the importance of that point seems diminished.

comment by PrawnOfFate · 2013-04-20T20:04:15.746Z · LW(p) · GW(p)

So, just to pick an example, IIRC Dan Dennett believes the philosophical study of consciousness (qualia, etc.) is fundamentally confused in more or less the same way Desrtopa claims of the philosophical study of ethics is.

Dennett only thinks the idea of qualia is confused. He has no problem with his own books on consciousness.

So under this formulation, if most of the institutional participants in the philosophical study of consciousness are intelligent, well-educated people, Dan Dennet is a crank?

No. He isn't dismissing a whole academic subject, or a sub-field. Just one idea.

Replies from: TheOtherDave
comment by TheOtherDave · 2013-04-20T20:33:27.255Z · LW(p) · GW(p)

What is Dennett's account for why philosophers of consciousness other than himself continue to think that a dismissable idea like qualia is worth continuing to discuss, even though he considers it closed?

comment by PrawnOfFate · 2013-04-20T19:37:54.008Z · LW(p) · GW(p)

Desrtopa doesn't think moral philosophy is uniformly nonsense, since Desrtopa thinks one of its well known claims, moral relativism, is true.

comment by Kawoomba · 2013-04-20T19:45:42.715Z · LW(p) · GW(p)

While going on tangents is a common and expected occurrence, each such tangent has a chance of steering/commandeering the original conversation. LW has a tendency of going meta too much, when actual object level discourse would have a higher content value.

While you were practically invited to indulge in the death-by-meta with the hook of "Are you aware that that is basically what every crank says about some other field?", we should be aware when leaving the object-level debating, and the consequences thereof. Especially since the lure can be strong:

When sufficiently meta, object-level disagreements may fizzle into cosmic/abstract insignificance, allowing for a peaceful pseudo-resolution, which ultimately just protects that which should be destroyed by the truth from being destroyed.

Such lures may be interpreted similarly to ad hominems: The latter try to drown out object-level disagreements by flinging shit until everyone's dirty, the former zoom out until everyone's dizzy floating in space, with vertigo. Same result to the actual debate. It's an effective device, and one usually embraced by someone who feels like object-level arguments no longer serve his/her goals.

Ironically, this very comment goes meta lamenting going meta.

comment by Desrtopa · 2013-04-20T19:16:09.105Z · LW(p) · GW(p)

I have no idea what you mean by that. I don't think value systems don't come into it, I just think they are not isolated from rationality. And I am sceptical that you could predict any higher-level phenomenon from "the ground up", whether its morality or mortgages.

I mean that value systems are a function of physically existing things, the way a 747 is a function of physically existing things, but we have no evidence suggesting that objective morality is an existing thing. We have standards by which we judge beauty, and we project those values onto the world, but the standards are in us, not outside of us. We can see, in reductionist terms, how the existence of ethical systems within beings, which would feel from the inside like the existence of an objective morality, would come about.

Create a reasoning engine that doesn't have those ethical systems built into it, and it would have no reason to care about them.

Where is it proven they can be discarded?

You can't build a tower on empty air. If a debate has been going on for hundreds of years, stretching back to an argument which rests on "this defies our moral intuitions, therefore it's wrong," and that was never addressed with "moral intuitions don't work that way," then the debate has failed to progress in a meaningful direction, much as a debate over whether a tree falling in an empty forest makes a sound has if nobody bothers to dissolve the question.

All of them.

That's not an example. Please provide an actual one.

Are you aware that that is basically what every crank says about some other field?

Sure, but it's also what philosophers say about each other, all the time. Wittgenstein condemned practically all his predecessors and peers as incompetent, and declared that he had solved nearly the entirety of philosophy. Philosophy as a field is full of people banging their heads on a wall at all those other idiots who just don't get it. "Most philosophers are incompetent, except for the ones who're sensible enough to see things my way," is a perfectly ordinary perspective among philosophers.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T19:51:46.398Z · LW(p) · GW(p)

I mean that value systems are a function of physically existing things, the way a 747 is a function of physically existing things, but we have no evidence suggesting that objective morality is an existing thing.

But I wans't saying that. I am arguing that moral claims truth values, that aren;t indexed to individuals or socieities. That epistemic claim can be justified by appeal to an ontoogy including Moral Objects, but that is not how I am justifying it: my argument is based on rationality, as I have said many times.

We have standards by which we judge beauty, and we project those values onto the world, but the standards are in us, not outside of us.

We have standards by which we jusdge the truth values of mathematical claims, and they are inside us too, and that doens't stop mathematics being objective. Relativism requires that truthvalues are indexed to us, that there is one truth for me and another for thee. Being located in us, or being operated by us are not sufficient criteria for being indexed to us.

We can see, in reductionist terms, how the existence of ethical systems within beings, which would feel from the inside like the existence of an objective morality, would come about.

We can see, in reductionistic terms, how the entities could converge on a unform set of truth values. There is nothing non reductionist about anything I have said. Reductionsm does not force one answer to metaethics.

reate a reasoning engine that doesn't have those ethical systems built into it, and it would have no reason to care about them.

Provide evidence that ethics is a whole separate modue, and not part of general reasoning ability.

You can't build a tower on empty air. If a debate has been going on for hundreds of years, stretching back to an argument which rests on "this defies our moral intuitions, therefore it's wrong," and that was never addressed with "moral intuitions don't work that way," then the debate has failed to progress in a meaningful direction, much as a debate over whether a tree falling in an empty forest makes a sound has if nobody bothers to dissolve the question.

Please explain why moral intuitions don't work that way.

Please provide some foundations for somethng that aren;t unjustofied by anything more foundationa.

That's not an example. Please provide an actual one

You can select one at random. obviously.

Sure, but it's also what philosophers say about each other, all the time.

No, philosophers don't regularly accuse each other of being incpompetent..just of being wrong. There's a difference.

Wittgenstein condemned practically all his predecessors and peers as incompetent, and declared that he had solved nearly the entirety of philosophy.

You are inferring a lot from one example.

Philosophy as a field is full of people banging their heads on a wall at all those other idiots who just don't get it. "Most philosophers are incompetent, except for the ones who're sensible enough to see things my way," is a perfectly ordinary perspective among philosophers.

Nope.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-20T20:29:38.966Z · LW(p) · GW(p)

But I wans't saying that. I am arguing that moral claims truth values, that aren;t indexed to individuals or socieities. That epistemic claim can be justified by appeal to an ontoogy including Moral Objects, but that is not how I am justifying it: my argument is based on rationality, as I have said many times.

I don't understand, can you rephrase this?

We have standards by which we jusdge the truth values of mathematical claims, and they are inside us too, and that doens't stop mathematics being objective. Relativism requires that truthvalues are indexed to us, that there is one truth for me and another for thee. Being located in us, or being operated by us are not sufficient criteria for being indexed to us.

The standards by which we judge the truth of mathematical claims are not just inside us. One object plus another object will continue to equal two objects whether or not there are any living beings to make that judgment. Math is not something we've created within ourselves, but something we've discovered and observed.

If our mathematical models ever stop being able to predict in advance the behavior of the universe, then we will have rather more reason to doubt that the math inside us is different from the math outside of us.

What evidence do we have that this is the case for morality?

Provide evidence that ethics is a whole separate modue, and not part of general reasoning ability.

My assertion is that, if we judge ethics as a rational system, innate values are among the axioms that the system is predicated on. You cannot prove the axioms of a system within that system, and an ethical system predicated on premises like "happiness is good" will not itself be able to prove the goodness of happiness.

While we could suppose that the axioms which our ethical systems are predicated on are objectively true, we have considerable reason to believe that we would have developed these axioms for adaptive reasons, even if there were no sense in which objective moral axioms exist, and we do not have evidence which suggests that objective, independently existing true moral axioms do exist.

Please explain why moral intuitions don't work that way.

People can be induced to strongly support opposing responses to the same moral dilemma, just by rephrasing it differently to trigger different heuristics. Our moral intuitions are incoherent.

Please provide some foundations for somethng that aren;t unjustofied by anything more foundationa.

I don't think I understand this, can you rephrase it?

You can select one at random. obviously.

I do not recall any creditable attempts, which places me in a disadvantaged position with respect to locating them. You're the one claiming that they're there at all, that's why I'm asking you to do it.

No, philosophers don't regularly accuse each other of being incpompetent..just of being wrong. There's a difference.

Philosophers don't usually accuse each other of being incompetent in their publications, because it's not conducive to getting other philosophers to regard their arguments dispassionately, and that sort of open accusation is generally frowned upon in academic circles whether one believes it or not. They do regularly accuse each other of being comprehensively wrong for their entire careers. In my personal conversations with philosophers (and I never considered myself to have really taken a class, or attended a lecture by a visitor, if I didn't speak with the person teaching it on a personal basis to probe their thoughts beyond the curriculum,) I observed a whole lot of frustration with philosophers who they think just don't get their arguments. It's unsurprising that people would tend to become so frustrated participating in a field that basically amounts to long running arguments extended over decades or centuries. Imagine the conversation we're having now going on for eighty years, and neither of us has changed our minds. If you didn't find my arguments convincing, and I hadn't budged in all that time, don't you'd think you'd start to suspect that I was particularly thick?

You are inferring a lot from one example.

I'm using an example illustrative of my experience.

Replies from: Nornagest, PrawnOfFate
comment by Nornagest · 2013-04-20T21:12:51.047Z · LW(p) · GW(p)

I don't understand, can you rephrase this?

Sounds to me like PrawnOfFate is saying that any sufficiently rational cognitive system will converge on a certain set of ethical goals as a consequence of its structure, i.e. that (human-style) ethics is a property that reliably emerges in anything capable of reason.

I'd say the existence of sociopathy among humans provides a pretty good counterargument to this (sociopaths can be pretty good at accomplishing their goals, so the pathology doesn't seem to be indicative of a flawed rationality), but at least the argument doesn't rely on counting fundamental particles of morality or something.

Replies from: Desrtopa, PrawnOfFate
comment by Desrtopa · 2013-04-20T21:20:39.536Z · LW(p) · GW(p)

I would say so also, but PrawnOfFate has already argued that sociopaths are subject to additional egocentric bias relative to normal people and thereby less rational. It seems to me that he's implicitly judging rationality by how well it leads to a particular body of ethics he already accepts, rather than how well it optimizes for potentially arbitrary values.

Replies from: Nornagest
comment by Nornagest · 2013-04-20T21:39:24.335Z · LW(p) · GW(p)

Well, I'm not a psychologist, but if someone asked me to name a pathology marked by unusual egocentric bias I'd point to NPD, not sociopathy.

That brings up some interesting questions concerning how we define rationality, though. Pathologies in psychology are defined in terms of interference with daily life, and the personality disorder spectrum in particular usually implies problems interacting with people or societies. That could imply either irreconcilable values or specific flaws in reasoning, but only the latter is irrational in the sense we usually use around here. Unfortunately, people are cognitively messy enough that the two are pretty hard to distinguish, particularly since so many human goals involve interaction with other people.

In any case, this might be a good time to taboo "rational".

comment by PrawnOfFate · 2013-04-22T13:52:03.179Z · LW(p) · GW(p)

Since no claim has a probability of 1.0, I only need to argue that a clear majority of rational minds converge.

comment by PrawnOfFate · 2013-04-22T13:50:35.382Z · LW(p) · GW(p)

The standards by which we judge the truth of mathematical claims are not just inside us.

How do we judge claims about transfinite numbers?

One object plus another object will continue to equal two objects whether or not there are any living beings to make that judgment. Math is not something we've created within ourselves, but something we've discovered and observed.

If our mathematical models ever stop being able to predict in advance the behavior of the universe, then we will have rather more reason to doubt that the math inside us is different from the math outside of us.

Mathematics isn't physics. Mathematicians prove theorems from axioms, not from experiments.

Provide evidence that ethics is a whole separate modue, and not part of general reasoning ability.

My assertion is that, if we judge ethics as a rational system, innate values are among the axioms that the system is predicated on.

Not necessarily. Eg, for utilitarians, values are just facts that are plugged into the metaethics to get concrete actions.

You cannot prove the axioms of a system within that system, and an ethical system predicated on premises like "happiness is good" will not itself be able to prove the goodness of happiness.

Metaethical systems usually have axioms like "Maximising utility is good".

While we could suppose that the axioms which our ethical systems are predicated on are objectively true, we have considerable reason to believe that we would have developed these axioms for adaptive reasons, even if there were no sense in which objective moral axioms exist, and we do not have evidence which suggests that objective, independently existing true moral axioms do exist.

I am not sure what you mean by "exist" here. Claims are objectively true if most rational minds converge on them. That doesn't require Objective Truth to float about in space here.

Please explain why moral intuitions don't work that way.

People can be induced to strongly support opposing responses to the same moral dilemma, just by rephrasing it differently to trigger different heuristics. Our moral intuitions are incoherent.

Does that mean we can;t use moral intuitions at all, or that they must be used with caution?

I don't think I understand this, can you rephrase it?

Philosphers talk about intuitions, because that is the term for something foundational that seems true, but can't be justified by anything more foundational. LessWrongians don't like intuitions, but don't see to be able to explain how to manage without them.

I do not recall any creditable attempts, which places me in a disadvantaged position with respect to locating them.

Did you post any comments explaining to the professional philosophers where they had gone wrong?

Imagine the conversation we're having now going on for eighty years, and neither of us has changed our minds. If you didn't find my arguments convincing, and I hadn't budged in all that time, don't you'd think you'd start to suspect that I was particularly thick?

I don;'t see the problem. Philosophical competence is largely about understanding the problem.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-22T14:33:45.119Z · LW(p) · GW(p)

Mathematics isn't physics. Mathematicians prove theorems from axioms, not from experiments.

Yes, but the fact that the universe itself seems to adhere to the logical systems by which we construct mathematics gives credence to the idea that the logical systems are fundamental, something we've discovered rather than producing. We judge claims about nonobserved mathematical constructs like transfinites according to those systems,

Metaethical systems usually have axioms like "Maximising utility is good".

But utility is a function of values. A paperclipper will produce utility according to different values than a human.

I am not sure what you mean by "exist" here. Claims are objectively true if most rational minds converge on them. That doesn't require Objective Truth to float about in space here.

Why would most rational minds converge on values? Most human minds converge on some values, but we share almost all our evolutionary history and brain structure. The fact that most humans converge on certain values is no more indicative of rational minds in general doing so than the fact that most humans have two hands is indicative of most possible intelligent species converging on having two hands.

Does that mean we can;t use moral intuitions at all, or that they must be used with caution?

It means we should be aware of what our intuitions are and what they've developed to be good for. Intuitions are evolved heuristics, not a priori truth generators.

Philosphers talk about intuitions, because that is the term for something foundational that seems true, but can't be justified by anything more foundational. LessWrongians don't like intuitions, but don't see to be able to explain how to manage without them.

It seems like you're equating intuitions with axioms here. We can (and should) recognize that our intuitions are frequently unhelpful at guiding us to he truth, without throwing out all axioms.

Did you post any comments explaining to the professional philosophers where they had gone wrong?

If I did, I don't remember them. I may have, I may have felt someone else adequately addressed them, I may not have felt it was worth the bother.

It seems to me that you're trying to foist onto me the effort of locating something which you were the one to testify was there in the first place.

I don;'t see the problem. Philosophical competence is largely about understanding the problem.

And philosophers frequently fall into the pattern of believing that other philosophers disagree with each other due to failure to understand the problems they're dealing with.

In any case, I reject the notion that dismissing large contingents of philosophers as lacking in competence is a valuable piece of evidence with respect t crankishness, and if you want to convince me that I am taking a crankish attitude, you'll need to offer some other evidence.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-22T21:24:56.897Z · LW(p) · GW(p)

Yes, but the fact that the universe itself seems to adhere to the logical systems by which we construct mathematics gives credence to the idea that the logical systems are fundamental, something we've discovered rather than producing. We judge claims about nonobserved mathematical constructs like transfinites according to those systems,

But claims about transfinities don't correspond directly to any object. Maths is "spun off" from other facts, on your view. So, by analogy, moral realism could be "spun off" without needing any Form of the Good to correspond to goodness.

Metaethical systems usually have axioms like "Maximising utility is good".

But utility is a function of values. A paperclipper will produce utility according to different values than a human.

You seem to be assumig that morality is about individual behaviour. A moral realist system like utiitarianism operates at the group level, and woud take paperclipper values into account along with all others. Utilitarianism doens't care what values are, it just sums or averages them.

Or perhaps you are making the objection that an entity woud need moral values to care about the preferences of others in the first place. That is addressed by, another kind of realism, the rationality-based kind, which starts from noting that rational agents have to have some value in common, because they are all rational.

Why would most rational minds converge on values?

a) they don't have to converge on preferences, since thing like utilitariansim are preference-neutral.

b) they already have to some extent because they are rational

Most human minds converge on some values, but we share almost all our evolutionary history and brain structure. The fact that most humans converge on certain values is no more indicative of rational minds in general doing so than the fact that most humans have two hands is indicative of most possible intelligent species converging on having two hands.

I was talking about rational minds converging on the moral claims, not on values.. Rational minds can converge on "maximise group utility" whilst what is utilitous varies considerably.

Philosphers talk about intuitions, because that is the term for something foundational that seems true, but can't be justified by anything more foundational. LessWrongians don't like intuitions, but don't see to be able to explain how to manage without them.

It seems like you're equating intuitions with axioms here.

Axioms are formal statements, intuitions are gut feelings tha are often used to justify axioms.

We can (and should) recognize that our intuitions are frequently unhelpful at guiding us to he truth, without throwing out all axioms.

There is another sense of "intuition" where someone feels that it's going to rain tomorrow or something. They're not the foundational kind.

And philosophers frequently fall into the pattern of believing that other philosophers disagree with each other due to failure to understand the problems they're dealing with.

So do they call for them to be fired?

Replies from: Desrtopa
comment by Desrtopa · 2013-04-22T21:40:32.108Z · LW(p) · GW(p)

But claims about transfinities don't correspond directly to any object. Maths is "spun off" from other facts, on your view. So, by analogy, moral realism could be "spun off" without needing any Form of the Good to correspond to goodness.

Spun off from what, and how?

You seem to be assumig that morality is about individual behaviour. A moral realist system like utiitarianism operates at the group level, and woud take paperclipper values into account along with all others. Utilitarianism doens't care what values are, it just sums or averages them.

Speaking as a utilitarian, yes, utilitarianism does care about what values are. If I value paperclips, I assign utility to paperclips, if I don't, I don't.

Or perhaps you are making the objection that an entity woud need moral values to care about the preferences of others in the first place. That is addressed by, another kind of realism, the rationality-based kind, which starts from noting that rational agents have to have some value in common, because they are all rational.

Why does their being rational demand that they have values in common? Being rational means that they necessarily share a common process, namely rationality, but that process can be used to optimize many different, mutually contradictory things. Why should their values converge?

I was talking about rational minds converging on the moral claims, not on values.. Rational minds can converge on "maximise group utility" whilst what is utilitous varies considerably.

So what if a paperclipper arrives at "maximize group utility," and the only relevant member of the group which shares its conception of utility is itself, and its only basis for measuring utility is paperclips? The fact that it shares the principle of maximizing utility doesn't demand any overlap of end-goal with other utility maximizers.

Axioms are formal statements, intuitions are gut feelings tha are often used to justify axioms.

But, as I've pointed out previously, intuitions are often unhelpful, or even actively misleading, with respect to locating the truth.

If our axioms are grounded in our intuitions, then entities which don't share our intuitions will not share our axioms.

So do they call for them to be fired?

No, but neither do I, so I don't see why that's relevant.

Replies from: Eliezer_Yudkowsky, PrawnOfFate
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-23T03:23:14.822Z · LW(p) · GW(p)

Designating PrawnOfFate a probable troll or sockpuppet. Suggest terminating discussion.

Replies from: Desrtopa, Bugmaster
comment by Desrtopa · 2013-04-23T04:58:56.093Z · LW(p) · GW(p)

Request accepted, I'm not sure if he's being deliberately obtuse, but I think this discussion probably would have borne fruit earlier if it were going to. I too often have difficulty stepping away from a discussion as soon as I think it's unlikely to be a productive use of my time.

comment by Bugmaster · 2013-04-23T04:46:47.017Z · LW(p) · GW(p)

What is your basis for the designation ? I am not arguing with your suggestion (I was leaning in the same direction myself), I'm just genuinely curious. In other words, why do you believe that PrawnOfFate is a troll, and not someone who is genuinely confused ?

Replies from: Eliezer_Yudkowsky, wedrifid
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-23T05:54:32.786Z · LW(p) · GW(p)

Combined behavior in other threads. Check the profile.

comment by wedrifid · 2013-04-23T09:45:22.246Z · LW(p) · GW(p)

In other words, why do you believe that PrawnOfFate is a troll, and not someone who is genuinely confused ?

"Troll" is a somewhat fuzzy label. Sometimes when I am wanting to be precise or polite and avoid any hint of Fundamental Attribution Error I will replace it with the rather clumsy or verbose "person who is exhibiting a pattern of behaviour which should not be fed". The difference between "Person who gets satisfaction from causing disruption" and "Person who is genuinely confused and is displaying an obnoxiously disruptive social attitude" is largely irrelevant (particularly when one has their Hansonian hat on).

If there was a word in popular use that meant "person likely to be disruptive and who should not be fed" that didn't make any assumptions or implications of the intent of the accused then that word would be preferable.

comment by PrawnOfFate · 2013-04-23T09:19:25.702Z · LW(p) · GW(p)

Spun off from what, and how?

I am not sure I can expalin that succintly at the moment. It is also hard to summarise how you get from counting apples to transfinite numbers.

Why does their being rational demand that they have values in common? Being rational means that they necessarily share a common process, namely rationality, but that process can be used to optimize many different, mutually contradictory things. Why should their values converge?

Rationality is not an automatic process, it is skill that has to be learnt and consciously applied. Individuals will only be rational if their values prompt them to. And rationality itself implies valuing certain things (lack of bias, non arbitrariness).

So what if a paperclipper arrives at "maximize group utility," and the only relevant member of the group which shares its conception of utility is itself, and its only basis for measuring utility is paperclips? The fact that it shares the principle of maximizing utility doesn't demand any overlap of end-goal with other utility maximizers.

Utilitarians want to maximise the utiity of their groups, not their own utility. They don;t have to believe the utlity of others is utilitous to them, they just need to feed facts about group utility into an aggregation function. And, using the same facts and same function, different utilitarians will converge. That's kind of the point.

But, as I've pointed out previously, intuitions are often unhelpful, or even actively misleading, with respect to locating the truth.

Compared to what? Remember, I am talking about foundational intuitions, the kind at the bottom of the stack. The empirical method of locating the truth rests on the intuition that the senses reveal a real external world. Which I share. But what proves it? That's the foundational issue.

comment by Paul Crowley (ciphergoth) · 2013-04-20T15:08:43.940Z · LW(p) · GW(p)

The question of moral realism is AFAICT orthogonal to the Orthogonality Thesis.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T15:31:50.920Z · LW(p) · GW(p)

A lot of people here would seem to disagree, since I keep hearing the objection that ethics is all about values, and values are nothing to do with rationality.

Replies from: ciphergoth
comment by Paul Crowley (ciphergoth) · 2013-04-21T12:55:11.711Z · LW(p) · GW(p)

Could you make the connection to what I said more explicit please? Thanks!

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-21T15:19:16.772Z · LW(p) · GW(p)

" values are nothing to do with rationality"=the Orthogonality Thesis, so it's a step in the argument.

Replies from: ciphergoth
comment by Paul Crowley (ciphergoth) · 2013-04-21T19:57:56.118Z · LW(p) · GW(p)

It feels to me like the Orthogonality Thesis is a fairly precise statement, and moral anti-realism is a harder to make precise but at least well understood statement, and "values are nothing to do with rationality" is something rather vague that could mean either of those things or something else.

comment by MugaSofer · 2013-04-23T11:29:51.861Z · LW(p) · GW(p)

I am getting the feeling that you're assuming there's something in the agent's code that says, "you can look at and change any line of code you want, except lines 12345..99999, because that's where your terminal goals are". Is that right ?

You can change that line, but it will result in you optimizing for something other than paperclips, resulting in less paperclips.

comment by MugaSofer · 2013-04-23T11:10:56.705Z · LW(p) · GW(p)

Suppose that you gained the power to both discern objective morality, and to alter your own source code. You use the former ability, and find that the basic morally correct principle is maximizing the suffering of sentient beings. Do you alter your source code to be in accordance with this?

I've never understood this argument.

It's like a slaveowner having a conversation with a time-traveler, and declaring that they don't want to be nice to slaves, so any proof they could show is by definition invalid.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-23T14:40:29.430Z · LW(p) · GW(p)

If the slaveowner is an ordinary human being, they already have values regarding how to treat people in their in-groups which they navigate around with respect to slaves by not treating them as in-group members. If they could be induced to see slaves as in-group members, they would probably become nicer to slaves whether they intended to or not (although I don't think it's necessarily the case that everyone who's sufficiently acculturated to slavery could be induced to see slaves as in-group members.)

If the agent has no preexisting values which can be called into service of the ethics they've being asked to adopt, I don't think that they could be induced to want to adopt them.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-25T15:59:39.612Z · LW(p) · GW(p)

Sure, but if there's an objective morality, it's inherently valuable, right? So you already value it. You just haven't realized it yet.

It gets even worse when people try to refute wireheading arguments with this. Or statements like "if it were moral to [bad thing], would you do it?"

Replies from: Desrtopa
comment by Desrtopa · 2013-04-25T22:02:59.460Z · LW(p) · GW(p)

What evidence would suggest that objective morality in such a sense could or does exist?

Replies from: MugaSofer
comment by MugaSofer · 2013-04-26T11:15:02.848Z · LW(p) · GW(p)

I'm not saying moral realism is coherent, merely that this objection isn't.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-26T12:54:49.352Z · LW(p) · GW(p)

I don't think it's true that if there's an objective morality, agents necessarily value it whether they realize it or not though. Why couldn't there be inherently immoral or amoral agents?

Replies from: MugaSofer
comment by MugaSofer · 2013-04-26T13:14:32.357Z · LW(p) · GW(p)

... because the whole point of an "objective" morality is that rational agents will update to believe they should follow it? Otherwise we might as easily be such "inherently immoral or amoral agents", and we wouldn't want to discover such objective "morality".

Replies from: Desrtopa
comment by Desrtopa · 2013-04-26T13:29:53.965Z · LW(p) · GW(p)

Well, if it turned out that something like "maximize suffering of intelligent agents" were written into the fabric of the universe, I think we'd have to conclude that we were inherently immoral agents.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-26T14:11:01.519Z · LW(p) · GW(p)

The same evidence that persuades you that we don't want to maximize suffering in real life is evidence that it wouldn't be, I guess.

Side note: I've never seen anyone try to defend the position that we should be maximizing suffering, whereas I've seen all sorts of eloquent and mutually contradictory defenses of more, um, traditional ethical frameworks.

comment by PrawnOfFate · 2013-04-20T02:20:39.388Z · LW(p) · GW(p)

A rational AI would use rationality. Amazing how that word keeps disappearing...on a website about...rationality.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-20T02:37:20.093Z · LW(p) · GW(p)

Elaborate. What rational process would it use to determine the silliness of its original objective?

comment by PrawnOfFate · 2013-04-23T11:40:59.748Z · LW(p) · GW(p)

However, both the pebble-sorters and myself share one key weakness: we cannot examine ourselves from the outside; we can't see our own source code.

Being able to read all you source code could be ultimate in self-reflection (absent Loeb's theorem), but it doens't follow that those who can't read their source-code can;t self reflect at all. It's just imperfect, like everything else.

comment by MugaSofer · 2013-04-23T11:07:03.021Z · LW(p) · GW(p)

"Objective".

comment by PrawnOfFate · 2013-04-20T01:16:48.784Z · LW(p) · GW(p)

As I was reading the article about the pebble-sorters, I couldn't help but think, "silly pebble-sorters, their values are so arbitrary and ultimately futile". This happened, of course, because I was observing them from the outside. If I was one of them, sorting pebbles would feel perfectly natural to me; and, in fact, I could not imagine a world in which pebble-sorting was not important. I get that.

This is about rational agents. If pebble sorters can't think of a non-arbitrary reason for sorting pebbles, they would recognise it a silly. Why not? Humans can spend years collecting stamps, or something, only to decide it is pointless.

However, both the pebble-sorters and myself share one key weakness: we cannot examine ourselves from the outside; we can't see our own source code. An AI, however, could

What...why...? Is there something special about silicon? Is it made from different quarks?

Replies from: Bugmaster, MugaSofer
comment by Bugmaster · 2013-04-20T01:50:00.171Z · LW(p) · GW(p)

This is about rational agents.

Being rational doesn't automatically make an agent able to read its own source code. Remember that, to the pebble-sorters, sorting pebbles is an axiomatically reasonable activity; it does not require justification. Only someone looking at them from the outside could evaluate it objectively.

What...why...? Is there something special about silicon?

Not at all; if you got some kind of a crazy biological implant that let you examine your own wetware, you could do it too. Silicon is just a convenient example.

Replies from: MugaSofer, PrawnOfFate
comment by MugaSofer · 2013-04-23T11:15:08.078Z · LW(p) · GW(p)

Not at all; if you got some kind of a crazy biological implant that let you examine your own wetware, you could do it too. Silicon is just a convenient example.

Humans can examine their own thinking. Not perfectly, because we aren't perfect. But we can do it, and indeed do so all the time. It's a major focus on this site, in fact.

comment by PrawnOfFate · 2013-04-20T01:59:30.244Z · LW(p) · GW(p)

Being rational doesn't automatically make an agent able to read its own source code. Remember that, to the pebble-sorters, sorting pebbles is an axiomatically reasonable activity;

You can define a pebblesorter as being unable to update its values, and I can point out that most rational agents won't be like that. Most rational agents won't have unupdateable values, because they will be messilly designed/evolved, and therefore will be capable of converging on an ethical system via their shared rationality.

Replies from: Bugmaster
comment by Bugmaster · 2013-04-20T02:58:08.015Z · LW(p) · GW(p)

Most rational agents won't have unupdateable values, because they will be messilly designed/evolved...

We are messily designed/evolved, and yet we do not have updatable goals or perfect introspection. I absolutely agree that some agents will have updatable goals, but I don't see how you can upgrade that to "most".

...and therefore will be capable of converging on an ethical system via their shared rationality.

How so ? Are you asserting that there exists an optimal ethical system that is independent of the actors' goals ? There may well be one, but I am not convinced of this, so you'll have to convince me.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T11:24:05.540Z · LW(p) · GW(p)

We are messily designed/evolved, and yet we do not have updatable goals or perfect introspection

We blatantly have updatable goals: people do not have the same goals at 5 as they do at 20 or 60.

I don't know why perfect introspection would be needed to have some ability to update.

.and therefore will be capable of converging on an ethical system via their shared rationality.

How so ? Are you asserting that there exists an optimal ethical system that is independent of the actors' goals ?

Yes, that's what this whole discussion is about.

Replies from: Bugmaster, MugaSofer
comment by Bugmaster · 2013-04-20T20:35:34.873Z · LW(p) · GW(p)

We blatantly have updatable goals: people do not have the same goals at 5 as they do at 20 or 60. I don't know why perfect introspection would be needed to have some ability to update.

Sorry, that was bad wording on my part; I should've said, "updatable terminal goals". I agree with what you said there.

How so ? Are you asserting that there exists an optimal ethical system that is independent of the actors' goals ?

Yes, that's what this whole discussion is about.

I don't feel confident enough in either "yes" or "no" answer, but I'm currently leaning toward "no". I am open to persuasion, though.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-22T11:02:56.436Z · LW(p) · GW(p)

I should've said, "updatable terminal goals".

You can make the evidence compatble with the theory of terminal values, but there is still no support for the theory of terminal values.

Replies from: Bugmaster
comment by Bugmaster · 2013-04-23T02:24:53.888Z · LW(p) · GW(p)

I personally don't know of any evidence in favor of terminal values, so I do agree with you there. Still, it makes a nice thought experiment: could we create an agent possessed of general intelligence and the ability to self-modify, and then hardcode it with terminal values ? My answer would be, "no", but I could be wrong.

That said, I don't believe that there exists any kind of a universally applicable moral system, either.

comment by MugaSofer · 2013-04-23T11:47:10.201Z · LW(p) · GW(p)

people do not have the same goals at 5 as they do at 20 or 60

Source?

They take different actions, sure, but it seems to me, based on childhood memories etc, that these are in the service of roughly the same goals. Have people, say, interviewed children and found they report differently?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-23T12:03:19.735Z · LW(p) · GW(p)

How many 5 year olds have the goal of Sitting Down WIth a Nice Cup of Tea?

Replies from: DaFranker, MugaSofer
comment by DaFranker · 2013-04-23T14:07:44.562Z · LW(p) · GW(p)

One less now that I'm not 5 years old anymore.

Could you please make a real argument? You're almost being logically rude.

comment by MugaSofer · 2013-04-23T14:00:43.339Z · LW(p) · GW(p)

Why do you think adults sit down with a nice cup of tea? What purpose does it serve?

comment by MugaSofer · 2013-04-23T11:14:00.299Z · LW(p) · GW(p)

This is about rational agents. If pebble sorters can't think of a non-arbitrary reason for sorting pebbles, they would recognise it a silly.

I'd use humans as a counterexample, but come to think, a lot of humans refuse to believe our goals could be arbitrary, and have developed many deeply stupid arguments that "prove" they're objective.

However, I'm inclined to think this is a flaw on the part of humans, not something rational.

comment by PrawnOfFate · 2013-04-20T01:00:28.585Z · LW(p) · GW(p)

One does not update terminal values, that's what makes them terminal.

Unicorns have horns...

Defining something something abstractly says nothing about its existence or likelihod. A neat division between terminal and abstract values could be implemented with sufficient effort, or could evolve with a low likelihood, but it is not a model of intelligence in general, and it is not likely just because messy solutions are likelier than neater ones. Actual and really existent horse-like beings are not going to acquire horns any time soon, no matter how clearly you define unicornhood.

Arguably, humans might not really have terminal values

Plausibly. You don;t now care about the same things you cared about when you were 10.

On what basis might a highly flexible paperclip optimizing program be persuaded that something else was more important than paperclips?

Show me one. Clippers are possible but not likely. I am not and never have said that Clippers would converge on the One True Ethics, I said that (super)intelligent, (super)rational agents would. The average SR-SI agent would not be a clipper for exactly the same reason that the average human is not an evil genius. There are no special rules for silicon!

Replies from: Desrtopa, MugaSofer
comment by Desrtopa · 2013-04-20T02:06:54.374Z · LW(p) · GW(p)

I'm noticing that you did not respond to my question of whether you've read No Universally Compelling Arguments and Sorting Pebbles Into Correct Heaps. I'd appreciate it if you would, because they're very directly relevant to the conversation, and I don't want to rehash the content when Eliezer has already gone to the trouble of putting them up where anyone can read them. If you already have, then we can proceed with that shared information, but if you're just going to ignore the links, how do I know you're going to bother giving due attention to anything I write in response?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T02:08:37.473Z · LW(p) · GW(p)

I'vre read them and you've been reading my response.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-20T02:34:55.063Z · LW(p) · GW(p)

Okay.

Plausibly. You don;t now care about the same things you cared about when you were 10.

I have different interests now than I did when I was ten, but that's not the same as having different terminal values.

Suppose a person doesn't support vegetarianism; they've never really given it much consideration, but they default to the assumption that eating meat doesn't cause much harm, and meat is tasty, so what's the big deal?

When they get older, they watch some videos on the conditions in which animals are raised for slaughter, read some studies on the neurology of livestock animals with respect to their ability to suffer, and decide that mainstream livestock farming does cause a lot of harm after all, and so they become a vegetarian.

This doesn't mean that their values have been altered at all. They've simply revised their behavior on new information with an application of the same values they already had. They started out caring about the suffering of sentient beings, and they ended up caring about the suffering of sentient beings, they just revised their beliefs about what actions that value should compel on the basis of other information.

To see whether person's values have changed, we would want to look, not at whether they endorse the same behaviors or factual beliefs that they used to, but whether their past self could relate to the reasons their present self has for believing and supporting the things they do now.

The average SR-SI agent would not be a clipper for exactly the same reason that the average human is not an evil genius.

The fact that humans are mostly not evil geniuses says next to nothing about the power of intelligence and rationality to converge on human standards of goodness. We all share almost all the same brainware. To a pebblesorter, humans would nearly all be evil geniuses, possessed of powerful intellects, yet totally bereft of a proper moral concern with sorting pebbles.

Many humans are sociopaths, and that slight deviation from normal human brainware results in people who cannot be argued into caring about other people for their own sakes. Nor can a sociopath argue a neurotypical person into becoming a sociopath.

If intelligence and rationality cause people to update their terminal values, why do sociopaths whose intelligence and rationality are normal to high by human standards (of which there are many) not update into being non-sociopaths, or vice-versa?

Replies from: MugaSofer, PrawnOfFate
comment by MugaSofer · 2013-04-23T11:26:06.185Z · LW(p) · GW(p)

Many humans are sociopaths, and that slight deviation from normal human brainware results in people who cannot be argued into caring about other people for their own sakes. Nor can a sociopath argue a neurotypical person into becoming a sociopath.

coughaynrandcough

Replies from: Desrtopa, OrphanWilde
comment by Desrtopa · 2013-04-23T14:42:37.287Z · LW(p) · GW(p)

There's a difference between being a sociopath and being a jerk. Sociopaths don't need to rationalize dicking other people over.

If Ayn Rand's works could actually turn formerly neurotypical people into sociopaths, that would be a hell of a find, and possibly spark a neuromedical breakthrough.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-25T15:56:20.072Z · LW(p) · GW(p)

That's beside the point, though. Just because two agents have incompatible values doesn't mean they can't be persuaded otherwise.

ETA: in other words, persuading a sociopath to act like they're ethical or vice versa is possible. It just doesn't rewire their terminal values.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-25T22:00:05.232Z · LW(p) · GW(p)

Sure, you can negotiate with an agent with conflicting values, but I don't think its beside the point.

You can get a sociopath to cooperate with non-sociopaths by making them trade off for things they do care about, or using coercive power. But Clippy doesn't have any concerns other than paperclips to trade off against its concern for paperclips, and we're not in a position to coerce Clippy, because Clippy is powerful enough to treat us as an obstacle to be destroyed. The fact that the non-sociopath majority can more or less keep the sociopath minority under control doesn't mean that we could persuade agents whose values deviate far from our own to accommodate us if we didn't have coercive power over them.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-26T11:18:15.537Z · LW(p) · GW(p)

Clippy is a superintelligence. Humans, neurotypical or no, are not.

I'm not saying it's necessarily rational for sociopaths to act moral or vice versa. I'm saying people can be (and have been) persuaded of this.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-26T12:59:54.803Z · LW(p) · GW(p)

Prawnoffate's point to begin with was that humans could and would change their fundamental values on new information about what is moral. I suggested sociopaths as an example of people who wouldn't change their values to conform to those of other people on the basis of argument or evidence, nor would ordinary humans change their fundamental values to a sociopath's.

If we've progressed to a discussion of whether it's possible to coerce less powerful agents into behaving in accordance with our values, I think we've departed from the context in which sociopaths were relevant in the first place.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-26T13:11:03.593Z · LW(p) · GW(p)

Oh, sorry, I wasn't disagreeing with you about that, just nitpicking your example. Should have made that clearer ;)

comment by OrphanWilde · 2013-04-23T16:23:31.603Z · LW(p) · GW(p)

Are you arguing Ayn Rand can argue sociopaths into caring about other people for their own sakes, or argue neurotypical people into becoming sociopaths?

(I could see both arguments, although as Desrtopa references, the latter seems unlikely. Maybe you could argue a neurotypical person into sociopathic-like behavior, which seems a weaker and more plausible claim.)

Replies from: MugaSofer
comment by MugaSofer · 2013-04-25T15:46:32.900Z · LW(p) · GW(p)

I could see both argument

Then that makes it twice as effective, doesn't it?

(Edited for clarity.)

comment by PrawnOfFate · 2013-04-20T11:48:48.109Z · LW(p) · GW(p)

I have different interests now than I did when I was ten, but that's not the same as having different terminal values.

You can construe the facts as being compatible with the theory of terminal values, but that doesn't actually support the theory of TVs.

To a pebblesorter, humans would nearly all be evil geniuses, possessed of powerful intellects, yet totally bereft of a proper moral concern with sorting pebbles.

Ethics is about regulating behaviour to take into account the preferences of others. I don't see how pebblesorting would count.

If intelligence and rationality cause people to update their terminal values, why do sociopaths whose intelligence and rationality are normal to high by human standards (of which there are many) not update into being non-sociopaths, or vice-versa?

Psychopathy is a strong egotistical bias.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-20T13:22:13.403Z · LW(p) · GW(p)

Ethics is about regulating behaviour to take into account the preferences of others. I don't see how pebblesorting would count.

How do you know that? Can you explain a process by which an SI-SR paperclipper could become convinced of this?

Psychopathy is a strong egotistical bias.

How can you you tell that psychopathy is an egotistical bias rather than non-psychopathy being an empathetic bias?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T13:28:10.361Z · LW(p) · GW(p)

How do you know that?

Much the same way as I understand the meanings of most words. Why is that a problem in this case.

How can you you tell that psychopathy is an egotistical bias rather than non-psychopathy being an empathetic bias?

Non psychopaths don't generally put other people above themselves--that is, they treat people equally, incuding themselevs.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-20T13:37:00.190Z · LW(p) · GW(p)

Much the same way as I understand the meanings of most words. Why is that a problem in this case.

"That's what it means by definition" wasn't much help to you when it came to terminal values, why do you think "that's what the word means" is useful here and not there? How do you determine that this word, and not that one, is an accurate description of a thing that exists?

Non psychopaths don't generally put other people above themselves--that is, they treat people equally, incuding themselevs.

This is not, in fact, true. Non-psychopaths routinely apply double standards to themselves and other people, and don't necessarily even realize they're doing it.

If we accept that it's true for the sake of an argument though, how do we know that they don't just have a strong egalitarian bias?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T13:43:32.870Z · LW(p) · GW(p)

How do you determine that this word, and not that one, is an accurate description of a thing that exists?

Are you saying ethical behavour doesn't exist on this planet, or that ethical behaviour as I have defined it doens't exist on this planet?

This is not, in fact, true. Non-psychopaths routinely apply double standards to themselves and other people, and don't necessarily even realize they're doing it.

OK. Non-psychopaths have a lesser degree of egotisitical bias. Does that prove they have some different bias? No. Does that prove an ideal rational and ethical agent would still have some bias from some point of view? No

This is not, in fact, true. Non-psychopaths routinely apply double standards to themselves and other people, and don't necessarily even realize they're doing it.

That's like saying they have a bias towards not having a bias.

Replies from: Desrtopa
comment by Desrtopa · 2013-04-20T13:53:45.868Z · LW(p) · GW(p)

Are you saying ethical behavour doesn't exist on this planet, or that ethical behaviour as I have defined it doens't exist on this planet?

I'm saying that ethical behavior as you have defined it is almost certainly not a universal psychological attractor. An SI-SR agent could look at humans and say "yep, this is by and large what humans think of as 'ethics,'" but that doesn't mean it would exert any sort of compulsion on it.

OK. Non-psychopaths have a lesser degree of egotisitical bias. Does that prove they have some different bias? No. Does that prove an ideal rational and ethical agent would still have some bias from some point of view? No

You not only haven't proven that psychopaths are the ones with an additional bias, you haven't even addressed the matter, you've just taken it for granted from the start.

How do you demonstrate that psychopaths have an egotistical bias, rather than non-psychopaths having an egalitarian bias, or rather than both of them having different value systems and pursuing them with equal degrees of rationality?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T14:01:24.118Z · LW(p) · GW(p)

I'm saying that ethical behavior as you have defined it is almost certainly not a universal psychological attractor.

I didn't say it was universal among all entities of all degrees of intelligence or rationality. I said there was a non neglible probability that agents of a certain level of rationality converging on an understanding of ethics.

An SI-SR agent could look at humans and say "yep, this is by and large what humans think of as 'ethics,'" but that doesn't mean it would exert any sort of compulsion on it.

"SR" stands to super rational. Rational agents find rational arguments rationally compelling. If rational arguments can be made for a certain understanding of ethics, they will be compelled by them.

You not only haven't proven that psychopaths are the ones with an additional bias,

Do you contest that psychopaths have more egotistical bias than the general population?

you've just taken it for granted from the start.

Yes. I thought it was something everyone knows.

rather than non-psychopaths having an egalitarian bias, o

it is absurd to characterise the practice of treating everyone the same as a form of bias.

Replies from: Desrtopa, TheOtherDave
comment by Desrtopa · 2013-04-20T14:17:18.993Z · LW(p) · GW(p)

I didn't say it was universal among all entities of all degrees of intelligence or rationality. I said there was a non neglible probability that agents of a certain level of rationality converging on an understanding of ethics.

Where does this non-negligible probability come from though? When I've asked you to provide any reason to suspect it, you've just said that as you're not arguing there's a high probability, there's no need for you to answer that.

"SR" stands to super rational. Rational agents find rational arguments rationally compelling. If rational arguments can be made for a certain understanding of ethics, they will be compelled by them.

I have been implicitly asking all along here, what basis do we have for suspecting at all that any sort of universally rationally compelling ethical arguments exist at all?

Do you contest that psychopaths have more egotistical bias than the general population?

Yes.

it is absurd to characterise the practice of treating everyone the same as a form of bias.

Why?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T17:20:23.993Z · LW(p) · GW(p)

Where does this non-negligible probability come from though?

Combining the probabilites of the steps of the argument.

I have been implicitly asking all along here, what basis do we have for suspecting at all that any sort of universally rationally compelling ethical arguments exist at all?

There are rationally compelling arguments.

Rationality probably universalisable since it is based on the avoidance of biases, incuding those regarding who and where your are.

There is nothing about ethics that makes it unseceptible to rational argument.

There are examples of rational argument about ethics, and of people being compelled by them.

Do you contest that psychopaths have more egotistical bias than the general population?

Yes.

That is an extraordinary claim, and the burden is on you to support it.

It is absurd to characterise the practice of treating everyone the same as a form of bias.

Why?

In the sense of "Nothing is a kind of something" or "atheism is a kind of religion".

Replies from: Desrtopa
comment by Desrtopa · 2013-04-20T18:44:28.218Z · LW(p) · GW(p)

There are rationally compelling arguments.

Rationality probably universalisable since it is based on the avoidance of biases, incuding those regarding who and where your are.

There is nothing about ethics that makes it unseceptible to rational argument.

There are examples of rational argument about ethics, and of people being compelled by them.

Rationality may be universalizable, but that doesn't mean ethics is.

If ethics are based on innate values extrapolated into systems of behavior according to their expected implications, then people will be susceptible to arguments regarding the expected implications of those beliefs, but not arguments regarding their innate values.

I would accept something like "if you accept that it's bad to make sentient beings suffer, you should oppose animal abuse" can be rationally argued for, but that doesn't mean that you can step back indefinitely and justify each premise behind it. How would you convince an entity which doesn't already believe it that it should care about happiness or suffering at all?

That is an extraordinary claim, and the burden is on you to support it.

I would claim the reverse, that saying that sociopathic people have additional egocentric bias is an extraordinary claim, and so I will ask you to support it, but of course, I am quite prepared to reciprocate by supporting my own claim.

It's much easier to subtract a heuristic from a developed mind by dysfunction than it is to add one. It is more likely as a prior that sociopaths are missing something that ordinary people possess, rather than having something that most people don't, and that something appears to be the brain functions normally concerned with empathy. It's not that they're more concerned with self interest than other people, but that they're less concerned with other people's interests.

Human brains are not "rationality+biases," so that a you could systematically subtract all the biases from a human brain and end up with perfect rationality. We are a bunch of cognitive adaptations, some of which are not at all in accordance with strict rationality, hacked together over our evolutionary history. So it makes little sense to judge humans with unusual neurology as being humans plus or minus additional biases, rather than being plus or minus additional functions or adaptations.

In the sense of "Nothing is a kind of something" or "atheism is a kind of religion".

Is it a bias to treat people differently from rocks?

Now, if we're going to categorize innate hardwired values, such as that which Clippy has for paperclips, as biases, then I would say "yes."

I don't think it makes sense to categorize such innate values as biases, and so I do not think that Clippy is "biased" compared to an ideally rational agent. Instrumental rationality is for pursuing agents' innate values. But if you think it takes bias to get you from not caring about paperclips to caring about paperclips, can you explain how, with no bias, you can get from not caring about anything, to caring about something?

If there were in fact some sort of objective morality, under which some people were much more valuable than others, then an ethical system which valued all people equally would be systematically biased in favor of the less valuable.

comment by TheOtherDave · 2013-04-20T16:32:58.696Z · LW(p) · GW(p)

it is absurd to characterise the practice of treating everyone the same as a form of bias.

Can you expand on what you mean by "absurd" here?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-20T17:08:42.621Z · LW(p) · GW(p)

In the sense of "Nothing is a kind of something" or "atheism is a kind of religion".

Replies from: TheOtherDave
comment by TheOtherDave · 2013-04-20T19:04:13.140Z · LW(p) · GW(p)

Hm.
OK.

So, I imagine the following conversation between two people (A and B):
A: It's absurd to say 'atheism is a kind of religion,'
B: Why?
A: Well, 'religion' is a word with an agreed-upon meaning, and it denotes a particular category of structures in the world, specifically those with properties X, Y, Z, etc. Atheism lacks those properties, so atheism is not a religion.
B: I agree, but that merely shows the claim is mistaken. Why is it absurd?
A: (thinks) Well, what I mean is that any mind capable of seriously considering the question 'Is atheism a religion?' should reach the same conclusion without significant difficulty. It's not just mistaken, it's obviously mistaken. And, more than that, I mean that to conclude instead that atheism is a religion is not just false, but the opposite of the truth... that is, it's blatantly mistaken.

Is A in the dialog above capturing something like what you mean?

If so, I disagree with your claim. It may be mistaken to characterize the practice of treating everyone the same as a form of bias, but it is not obviously mistaken or blatantly mistaken. In fact, I'm not sure it's mistaken at all, though if it is a bias, it's one I endorse among humans in a lot of contexts.

So, terminology aside, I guess the question I'm really asking is: how would I conclude that treating everyone the same (as opposed to treating different people differently) is not actually a bias, given that this is not obvious to me?

comment by MugaSofer · 2013-04-23T11:23:41.677Z · LW(p) · GW(p)

Plausibly. You don;t now care about the same things you cared about when you were 10.

Are we talking sweeties here? Because that seems more like lack of foresight than value drift. Or are we talking puberty? That seems more like new options becoming available.

I am not and never have said that Clippers would converge on the One True Ethics, I said that (super)intelligent, (super)rational agents would.

You should really start qualifying that with "most actual" if you don't want people to interpret it as applying to all possible (superintelligent) minds.

comment by MugaSofer · 2013-04-23T11:05:32.830Z · LW(p) · GW(p)

But you're talking about parts of mindspace other than ours, right? The Superhappies are strikingly similar to us, but they still choose the superhappiest values, not the right ones.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-23T11:18:54.123Z · LW(p) · GW(p)

I don't require their values to converge, I require them to accept the truths of certain claims. This happens in real life. People say "I don't like X, but I respect your right to do it". The first part says X is a disvalue, the second is an override coming from rationality.

Replies from: nshepperd, MugaSofer
comment by nshepperd · 2013-04-23T15:46:00.896Z · LW(p) · GW(p)

This is where you are confused. Almost certainly it is not the only confusion. But here is one:

Values are not claims. Goals are not propositions. Dynamics are not beliefs.

A machine that maximises paperclips can believe all true propositions in the world, and go on maximising paperclips. Nothing compels it to act any differently. You expect that rational agents will eventually derive the true theorems of morality. Yes, they will. Along with the true theorems of everything else. It won't change their behaviour, unless they are built so as to send those actions identified as moral to the action system.

If you don't believe me, I can only suggest you study AI (Thrun & Norvig) and/or the metaethics sequence until you do. (I mean really study. As if you were learning particle physics. It seems the usual metaethical confusions are quite resilient; in most peoples' cases I wouldn't expect them to vanish without actually thinking carefully about the data presented.) And, well, don't expect to learn too much from off-the-cuff comments here.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-23T17:40:13.199Z · LW(p) · GW(p)

A machine that maximises paperclips can believe all true propositions in the world, and go on maximising paperclips. Nothing compels it to act any differently. You expect that rational agents will eventually derive the true theorems of morality. Yes, they will.

Well, that justifies moral realism.

Along with the true theorems of everything else. It won't change their behaviour, unless they are built so as to send those actions identified as moral to the action system.

...or its an emergent feature, or they can update into something that works that way. You are tacitly assuming that you clipper is barely an AI at all...that is just has certain functions it performs blindly because its built that way. But a supersmart, uper-rational clipper has to be able to update. By hypothesis, clippers have certain functionalities walled off from update. People are messilly designed and unlikely to work that way. So are likely AIs and aliens.

Only rational agents, not all mindful agents, will have what it takes to derive objective moral truths. They don't need to converge on all their values to converge on all their moral truths, because ratioanity can tell you that a moral claim is true even if it is not in your (other) interests. Individuals can value rationality, and that valuation can override other valuations.

Only rational agents, not all mindful agents, will have what it takes to derive objective moral truths. The further claim that agents will be motivated to do derive moral truths., and to act on them, requires a further criterion. Morality is about regulating behaviour in a society, So only social rational agents will have motivation to update. Again, they do not have to converge on values beyond the shared value of sociality.

Replies from: nshepperd, DaFranker
comment by nshepperd · 2013-04-24T01:37:08.135Z · LW(p) · GW(p)

emergent

The Futility of Emergence

By hypothesis, clippers have certain functionalities walled off from update.

A paperclipper no more has a wall stopping it from updating into morality than my laptop has a wall stopping it from talking to me. My laptop doesn't talk to me because I didn't program it to. You do not update into pushing pebbles into prime-numbered heaps because you're not programmed to do so.

Does a stone roll uphill on a whim?

Perhaps you should study Reductionism first.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-24T01:44:26.816Z · LW(p) · GW(p)

The Futility of Emergence

"Emergent" in this context means "not explicitly programmed in". There are robust examples.

A paperclipper no more has a wall stopping it from updating into morality than my laptop has a wall stopping it from talking to me.

Your laptop cannot talk to you because the natural language is an unsolved problem.

Does a stone roll uphill on a whim?

Not wanting to do something is not the slightest guarantee of not actually doing it.f

An AI can update its values because value drift is an unsolved problem

Clippers can't update their values by definition, but you can't define anything into existence or statistical significance.

You do not update into pushing pebbles into prime-numbered heaps because you're not programmed to do so.

Not programmed to, or programmed not to? If you can code up a solution to value drift, lets see it. Otherwise, note that Life programmes can update to implement glider generators without being "programmed to".

Replies from: Nornagest
comment by Nornagest · 2013-04-24T02:19:21.549Z · LW(p) · GW(p)

Not programmed to, or programmed not to? If you can code up a solution to value drift, lets see it. Otherwise, note that Life programmes can update to implement glider generators without being "programmed to".

...with extremely low probability. It's far more likely that the Life field will stabilize around some relatively boring state, empty or with a few simple stable patterns. Similarly, a system subject to value drift seems likely to converge on boring attractors in value space (like wireheading, which indeed has turned out to be a problem with even weak self-modifying AI) rather than stable complex value systems. Paperclippism is not a boring attractor in this context, and a working fully reflective Clippy would need a solution to value drift, but humanlike values are not obviously so, either.

Replies from: PrawnOfFate, PrawnOfFate
comment by PrawnOfFate · 2013-04-25T12:34:28.394Z · LW(p) · GW(p)

I'm increasingly baffled as to why AI is always brought in to discussions of metaethics. Societies of rational agents need ethics to regulate their conduct. Out AIs aren't sophisticated enough to live in their own socieities. A wireheading AI isn't even going to be able to survive "in the wild". If you could build an artificial society of AI, then the questions of whether they spontaneously evolved ethics would be a very interesting and relevant datum. But AIs as we know them aren't good models for the kinds of entities to which morality is relevant. And Clippy is particularly exceptional example of an AI. So why do people keep saying "Ah, but Clippy..."...?

Replies from: Nornagest, MugaSofer
comment by Nornagest · 2013-04-25T18:45:38.666Z · LW(p) · GW(p)

And Clippy is particularly exceptional example of an AI. So why do people keep saying "Ah, but Clippy..."...?

Well, in this case it's because the post I was responding to mentioned Clippy a couple of times, so I thought it'd be worthwhile to mention how the little bugger fits into the overall picture of value stability. It's indeed somewhat tangential to the main point I was trying to make; paperclippers don't have anything to do with value drift (they're an example of a different failure mode in artificial ethics) and they're unlikely to evolve from a changing value system.

comment by MugaSofer · 2013-04-25T12:41:40.872Z · LW(p) · GW(p)

Key word here being "societies". That is, not singletons. A lot of the discussion on metaethics here is implicitly aimed at FAI.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-25T12:56:33.819Z · LW(p) · GW(p)

Sorry..did you mean FAI is about societies, or FAI is about singletons?

But if ethics does emerge as an organisational principle in socieities, that's all you need for FAI. You don't even to to worry about one sociopathic AI turning unfriendly, because the majority will be able to restrain it.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-25T13:45:18.590Z · LW(p) · GW(p)

FAI is about singletons, because the first one to foom wins, is the idea.

ETA: also, rational agents may be ethical in societies, but there's no advantage to being an ethical singleton.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-25T14:01:28.747Z · LW(p) · GW(p)

UFAI is about singletons. If you have an AI society whose members compare notes and share information -- which ins isntrumentally useful for them anyway -- your reduce the probability of singleton fooming.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-25T16:13:23.498Z · LW(p) · GW(p)

Any agent that fooms becomes a singleton. Thus, it doesn't matter if they acted nice while in a society; all that matters is whether they act nice as a singleton.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-25T16:15:20.114Z · LW(p) · GW(p)

I don't get it: any agent that fooms becomes superintelligent. It's values don't necessarily change at all, nor does its connection to its society.

Replies from: Randaly
comment by Randaly · 2013-04-25T16:34:41.705Z · LW(p) · GW(p)

An agent in a society is unable to force its values on the society; it needs to cooperate with the rest of society. A singleton is able to force its values on the rest of society.

comment by PrawnOfFate · 2013-04-24T02:23:49.066Z · LW(p) · GW(p)

At last, an interesting reply!

comment by DaFranker · 2013-04-23T18:25:30.914Z · LW(p) · GW(p)

Other key problem:

But a supersmart, uper-rational clipper has to be able to update.

has to be able to update

"update"

Please unpack this and describe precisely, in algorithmic terms that I could read and write as a computer program given unlimited time and effort, this "ability to update" which you are referring to.

I suspect that you are attributing Magical Powers From The Beyond to the word "update", and forgetting to consider that the ability to self-modify does not imply active actions to self-modify in any one particular way that unrelated data bits say would be "better", unless the action code explicitly looks for said data bits.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-23T18:39:56.372Z · LW(p) · GW(p)

It's uncontrovesial that rational agents need to update, and that AIs need to self-modify. The claim that values are in either case insulated from updates is the extraordinary one. The Cipper theory tells you that you could build something like that if you were crazy enough. Since Clippers are contrived, nothing can be inferred from them about typical agents. People are messy, and can accidentally update their values when trying to do something else, For instance, LukeProg updated to "atheist" after studying Christian apologetics for the opposite reason.

Replies from: TheOtherDave, DaFranker
comment by TheOtherDave · 2013-04-23T19:59:11.366Z · LW(p) · GW(p)

Yes, value drift is the typical state for minds in our experience.

Building a committed Clipper that cannot accidentally update its values when trying to do something else is only possible after the problem of value drift has been solved. A system that experiences value drift isn't a reliable Clipper, isn't a reliable good-thing-doer, isn't reliable at all.

Next.

comment by DaFranker · 2013-04-23T19:32:46.895Z · LW(p) · GW(p)

It's uncontrovesial that rational agents need to update, and that AIs need to self-modify. The claim that values are in either case insulated from updates is the extraordinary one.

I never claimed that it was controversial, nor that AIs didn't need to self-modify, nor that values are exempt.

I'm claiming that updates and self modification do not imply a change of behavior towards behavior desired by humans.

I can build a small toy program to illustrate, if that would help.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-23T19:39:41.601Z · LW(p) · GW(p)

I am not suggesting that human ethics is coincidentally universal ethics. I am suggesting that if neither moral realism nor relativism is initially discarded, one can eventually arrive at a compromise position where rational agents in a particular context arrive at a non arbitrary ethics which is appropriate to that context.

comment by MugaSofer · 2013-04-23T13:55:21.029Z · LW(p) · GW(p)

... why do you think people say "I don't like X, but I respect your right to do it"?

comment by PrawnOfFate · 2013-04-19T18:00:13.284Z · LW(p) · GW(p)

Well then, a universally correct solution based on axioms which can be chosen by the agents is a contradiction in and of itself.

if its based on arbitrary axioms, that would be a problem, but I have already argued that the axiom choice would not be arbitrary.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-19T18:11:04.205Z · LW(p) · GW(p)

I presume that you take your particular ethical system (or a variant thereof) to be the one that every alien, AI and human should adopt.

Ok, so why? Why can the function ethics: actions -> degree of goodness, or however else you choose the domain, not be modified? Where's your case?

Edit: What basis would convince not one, but every conceivable superintelligence of that hypothetical choice of axioms being correct? (They wouldn't all "cancel out" if choosing different axioms, that in itself would falsify the ethical system proposed by a lowly human as being universally correct.)

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-19T18:25:29.940Z · LW(p) · GW(p)

Well then, a universally correct solution based on axioms which can be chosen by the agents is a contradiction in and of itself.

I have not put forward an object-level ethical system, and I have explained why I do not need to. Physical realism does not imply that my physics is correct, metaethical realism does not imply that my ethics is the one true theory.

Ok, so why? Why can the function ethics: actions -> degree of goodness, or however else you choose the domain, not be modified?

Because ethics needs to regulate behaviour -- that is its functional role -- and could not if individuals could justify any behaviour by re arranging action->goodness mappings.

: What basis would convince not one, but every conceivable superintelligence of that hypothetical choice of axioms being correct? (

Their optimally satisfying the constraints on ethical axioms arising from the functional role of ethics.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-19T19:09:13.850Z · LW(p) · GW(p)

Well then, a universally correct solution based on axioms which can be chosen by the agents is a contradiction in and of itself.

I have not put forward an object-level ethical system, and I have explained why I do not need to. Physical realism does not imply that my physics is correct, metaethical realism does not imply that my ethics is the one true theory.

That doesn't actually answer the quoted point. Perhaps you meant to respond to this:

I presume that you take your particular ethical system (or a variant thereof) to be the one that every alien, AI and human should adopt.

... which is, in fact, refuted by your statement.

Because ethics needs to regulate behaviour -- that is its functional role -- and could not if individuals could justify any behaviour by re arranging action->goodness mappings.

... which Kawoomba believes they can, AFAICT.

Their optimally satisfying the constraints on ethical axioms arising from the functional role of ethics.

Could you unpack this a little? I think I see what you're driving at, but I'm not sure.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-19T21:09:40.546Z · LW(p) · GW(p)

Perhaps you meant to respond to this:

Yes, I did, thanks.

" if individuals could justify any behaviour"

which Kawoomba believes they can, AFAICT.

Then what about the second half of the argument? If individuals can "ethically" justify any behaviour, then does or does not such "ethics" completely fail in its essential role of regulating behaviour? Because anyone can do anything, and conjure up a justification after the fact by shifting their "frame"? A chocolate "teapot" is no teapot, non-regulative "ethics" is no ethics...

Could you unpack this a little?

Not now.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-19T22:54:14.852Z · LW(p) · GW(p)

Ah, but Kawoomba doesn't expect ethics to regulate other people, because he thinks everyone has incompatible goals. Thus ethics serves purely to define your goals.

Which, honestly, should simply be called "goals", not "ethics", but there you go.

Replies from: Kawoomba, PrawnOfFate
comment by Kawoomba · 2013-04-23T09:53:36.817Z · LW(p) · GW(p)

Yea, honestly I've never seen the exact distinction between goals which have an ethics-rating, and goals which do not. I understand that humans share many ethical intuitions, which isn't surprising given our similar hardware. Also, that it may be possible to define some axioms for "medieval Han Chinese ethics" (or some subset thereof), and then say we have an objectively correct model of their specific ethical code. About the shared intuitions amongst most humans, those could be e.g. "murdering your parents is wrong" (not even "murder is wrong", since that varies across cultures and circumstances). I'd still call those systems different, just as different cars can have the same type of engine.

Also, I understand that different alien cultures, using different "ethical axioms", or whatever they base their goals on, do not invalidate the medieval Han Chinese axioms, they merely use different ones.

My problem with "objectively correct ethics for all rational agents" is, you could say, where the compellingness of any particular system comes in. There is reason to believe an agent such as Clippy could not exist (edit: i.e., it probably could exist), and its very existence would contradict some "'rational' corresponds to a fixed set of ethics" rule. If someone would say "well, Clippy isn't really rational then", that would just be torturously warping the definition of "rational actor" to "must also believe in some specific set of ethical rules".

If I remember correctly, you say at least for humans there is a common ethical basis which we should adopt (correct me otherwise). I guess I see more variance and differences where you see common elements, especially going in the future. Should some bionically enhanced human, or an upload on a spacestation which doesn't even have parents, still share all the same rules for "good" and "bad" as an Amazon tribe living in an enclosed reservation? "Human civilization" is more of a loose umbrella term, and while there certainly can be general principles which some still share, I doubt there's that much in common in the ethical codex of an African child soldier and Donald Trump.

Replies from: PrawnOfFate, MugaSofer
comment by PrawnOfFate · 2013-04-23T10:21:56.927Z · LW(p) · GW(p)

Yea, honestly I've never seen the exact distinction between goals which have an ethics-rating, and goals which do not

A number of criteria have been put forward. For instance, do as you would be done by. If you don't want to be murdered, murder is not an ethical goal.

My problem with "objectively correct ethics for all rational agents" is, you could say, where the compellingness of any particular system comes in. There is reason to believe an agent such as Clippy could exist, and its very existence would contradict some "'rational' corresponds to a fixed set of ethics" rule. If someone would say "well, Clippy isn't really rational then", that would just be torturously warping the definition of "rational actor" to "must also believe in some specific set of ethical rules".

The argument is not that rational agents (for some vaue of "rational") must believe in some rules, it is rather that they must not adopt arbitrary goals. Also, the argument only requires a statistical majority of rational agents to converge, because of the P<1.0 thing.

Should some bionically enhanced human, or an upload on a spacestation which doesn't even have parents, still share all the same rules for "good" and "bad" as an Amazon tribe living in an enclosed reservation?

Maybe not. The important thing is that variations in ethics should not be arbitrary--they should be systematically related to variations in circumstances.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-23T10:38:06.173Z · LW(p) · GW(p)

I'm not disputing that there are goals/ethics which may be best suited to take humanity along a certain trajectory, towards a previously defined goal (space exploration!). Given a different predefined goal, the optimal path there would often be different. Say, ruthless exploitation may have certain advantages in empire building, under certain circumstances.

The Categorical Imperative in all its variants may be a decent system for humans (not that anyone really uses it).

But is the justification for its global applicability that "if everyone lived by that rule, average happiness would be maximized"? That (or any other such consideration) itself is not a mandatory goal, but a chosen one. Choosing different criteria to maximize (e.g. noone less happy than x) would yield different rules, e.g. different from the Categorical Imperative. If you find yourself to be the worshipped god-king in some ancient Mesopotanian culture, there may be many more effective ways to make yourself happy, other than the Categorical Imperative. How can it still be said to be "correct"/optimal for the king, then?

So I'm not saying there aren't useful ethical system (as judged in relation to some predefined course), but that because those various ultimate goals of various rational agents (happiness, paperclips, replicating yourself all over the universe) and associated optimal ethics vary, there cannot be one system that optimizes for all conceivable goals.

My argument against moral realism and assorted is that if you had an axiomatic system from which it followed that strawberry is the best flavor of ice cream, but other agents which are just as intelligent with just as much optimizing power could use different axiomatic systems leading to different conclusions, how could one such system possibly be taken to be globally correct and compelling-to-adopt across agents with different goals?

Gandhi wouldn't take a pill which may transform him into a murderer. Clippy would not willingly modify itself such that suddenly it had different goals. Once you've taken a rational agent apart and know its goals and, as a component, its ethical subroutines, there is no further "core spark" which really yearns to adopt the Categorical Imperative. Clippy may choose to use it, for a time, if it serves its ultimate goals. But any given ethical code will never be optimal for arbitrary goals, in perpetuity (proof by example). When then would a particular code following from particular axioms be adopted by all rational agents?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-23T10:50:13.361Z · LW(p) · GW(p)

But is the justification for its global applicability that "if everyone lived by that rule, average happiness would be maximized"?

Well, not, that's not Kant's justification!

That (or any other such consideration) itself is not a mandatory goal, but a chosen one.

Why would a rational agent choose unhappiness?

If you find yourself to be the worshipped god-king in some ancient Mesopotanian culture, there may be many more effective ways to make yourself happy, other than the Categorical Imperative.

Yes, but that wouldn't count as ethics. You wouldn't want a Universal Law that one guy gets the harem, and everyone else is a slave, because you wouldn't want to be a slave, and you probably would be. This is brought out in Rawls' version of Kantian ethics: you pretend to yourself that you are behind a veil that prevents you knowing what role in society you are going to have, and choose rules that you would want to have if you were to enter society at random.

My argument against moral realism and assorted is that if you had an axiomatic system from which it followed that strawberry is the best flavor of ice cream, but other agents which are just as intelligent with just as much optimizing power could use different axiomatic systems leading to different conclusions,

You don't have object-level stuff like ice cream or paperclips in your axioms (maxims), you have abstract stuff, like the Categorical Imperative. You then arrive at object level ethics by plugging in details of actual circumstances and values. These will vary, but not in an arbitrary way, as is the disadvantage of anything-goes relativism.

how could one such system possibly be taken to be globally correct and compelling-to-adopt across agents with different goals?

The idea is that things like the CI have rational appeal.

Once you've taken a rational agent apart and know its goals and, as a component, its ethical subroutines, there is no further "core spark" which really yearns to adopt the Categorical Imperative.

Rational agents will converge on a number of things because they are rational. None of them will think 2+2-=5.

Replies from: Kawoomba, MugaSofer
comment by Kawoomba · 2013-04-23T11:02:40.693Z · LW(p) · GW(p)

Scenario:

1) You wake up in a bright box of light, no memories. You are told you'll presently be born into an Absolute monarchy, your role randomly chosen. You may choose any moral principles that should govern that society. The Categorical Imperative would on average give you the best result.

2) You are the monarch in that society, you do not need to guess which role you're being born into, you have that information. You don't need to make all the slaves happy to help your goals, you can just maximize your goals directly. You may choose any moral principle you want to govern your actions. The Categorical Imperative would not give you the best result.

A different scenario: Clippy and Anti-Clippy sit in a room. Why can they not agree on epistemic facts about the most accurate laws of physics and other Aumann-mandated agreements, yet then go out and each optimize/reshape the world according to their own goals? Why would that make them not rational?

Lastly, whatever Kant's justification, why can you not optimize for a different principle - peak happiness versus average happiness, what makes any particular justifying principle correct across all - rational - agents. Here come my algae!

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-23T12:14:54.538Z · LW(p) · GW(p)

You are the monarch in that society, you do not need to guess which role you're being born into, you have that information. You don't need to make all the slaves happy to help your goals, you can just maximize your goals directly. You may choose any moral principle you want to govern your actions. The Categorical Imperative would not give you the best result.

For what value of "best"? If the CI is the correct theory of morality, it will necessarily give your the morally best result. Maybe your complaint is that it wouldn't maximise your personally utility. But I don't see why you would expect that. Things like utilitarianism that seek to maximise group utility, don't promise to make everyone blissfully happy individually. Some will lose out.

A different scenario: Clippy and Anti-Clippy sit in a room. Why can they not agree on epistemic facts about the most accurate laws of physics and other Aumann-mandated agreements, yet then go out and each optimize/reshape the world according to their own goals? Why would that make them not rational?

It would be irrational for Clippy to sing up to an agreement with Beady according to which Beady gets to turn Clippy and all his clips into beads. It is irrational for agents to sign up to anyhting which is not in their interests, and it is not in their interests to have no contract at all. So rational agents, even if they do not converge on all their goals, will negotiate contracts that minimise their disutility Clippy and Beady might take half the universe each.

Lastly, whatever Kant's justification, why can you not optimize for a different principle - peak happiness versus average happiness, what makes any particular justifying principle correct across all - rational - agents.

If you think RAs can converge on an ultimately correct theory of physics (which we don't have), what is to stop them converging on the correct theory of morality, which we also don't have?

Replies from: Kawoomba, MugaSofer
comment by Kawoomba · 2013-04-23T12:43:49.676Z · LW(p) · GW(p)

Some will lose out.

Not very rational for those to adopt a losing strategy (from their point of view), is it? Especially since they shouldn't reason from a point of "I could be the king". They aren't, and they know that. No reason to ignore that information, unless they believe in some universal reincarnation or somesuch.

It is irrational for agents to sign up to anyhting which is not in their [added: current] interests

Yes. Which is why rational agents wouldn't just go and change/compromise their terminal values, or their ethical judgements (=no convergence).

what is to stop them converging on the correct theory of morality, which we also don't have?

Starting out with different interests. A strong clippy accommodating a weak beady wouldn't be in its best self-interest. It could just employ a version of morality which is based on some tweaked axioms, yielding different results.

There are possibly good reasons for us as a race to aspire to working together. There are none for a domineering Clippy to take our interests into account, yielding to any supposedly "correct" morality would strictly damage its own interests.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-23T15:06:08.940Z · LW(p) · GW(p)

Not very rational for those to adopt a losing strategy (from their point of view), is it? Especially since they shouldn't reason from a point of "I could be the king". They aren't, and they know that. No reason to ignore that information, unless they believe in some universal reincarnation or somesuch.

Someone who adopts the "I don;t like X, but I respect peoples right to do it" approach is sacrificing some of their values to their evaluation of rationality and fairness. They would not do that if their rationality did not outweigh other values, But they are not having all their values maximally satisfied, so in that sense they are losing out.

Yes. Which is why rational agents wouldn't just go and change/compromise their terminal values, or their ethical judgements (=no convergence).

There's no evidence of terminal values. Judgements can be updated without changing values.

Starting out with different interests. A strong clippy accomodating a weak beady wouldn't be in its best self-interest. It could just employ a version of morality which is based on some tweaked axioms, yielding different results.

Not all agents are interested in physics or maths. Doesn't stop their claims being objetive.

comment by MugaSofer · 2013-04-23T13:23:58.212Z · LW(p) · GW(p)

It would be irrational for Clippy to sing up to an agreement with Beady according to which Beady gets to turn Clippy and all his clips into beads. It is irrational for agents to sign up to anyhting which is not in their interests, and it is not in their interests to have no contract at all. So rational agents, even if they do not converge on all their goals, will negotiate contracts that minimise their disutility Clippy and Beady might take half the universe each.

Not Beady, Anti-Clippy: an agent that is the precise opposite of Clippy. It wants to minimize the number of paperclips.

comment by MugaSofer · 2013-04-23T13:22:57.611Z · LW(p) · GW(p)

Yes, but that wouldn't count as ethics. You wouldn't want a Universal Law that one guy gets the harem, and everyone else is a slave, because you wouldn't want to be a slave, and you probably would be.

If there are a lot of similar agents in similar positions, Kantian ethics works, no matter what their goals. For example, theft may appear to have positive expected value - assuming you're selfish - but it has positive expected value for lots of people, and if they all stole the economy would collapse.

OTOH, if you are in an unusual position, the Categorical Imperative only has force if you take it as axiomatic.

This is brought out in Rawls' version of Kantian ethics: you pretend to yourself that you are behind a veil that prevents you knowing what role in society you are going to have, and choose rules that you would want to have if you were to enter society at random.

That's not a version of Kantian ethics, it's a hack for designing a society without privileging yourself. If you're selfish, it's a bad idea.

comment by MugaSofer · 2013-04-23T14:02:56.022Z · LW(p) · GW(p)

Kawoomba, maybe it would be better for you to think in terms of ethics along the lines of Kant's Categorical Imperative, or social contract theory; ways for agents with different goals to co-operate.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-23T14:13:43.091Z · LW(p) · GW(p)

Wouldn't that presuppose that "cooperation is the source/the sine qua non of all good"?

Sure, we can redefine some version of ethics in such a cooperative light, and then conclude that many agents don't give a hoot about such ethics, or regard it in the cold, hard terms of game theory, e.g. negotiating/extortion strategies only.

Judging actions as "good" or "bad" doesn't prima facie depend entirely on cooperation, the good of your race, or whatever. For example, if you were a part of a planet-eating race, consuming all matter/life in its path - while being very friendly amongst themselves - couldn't it be considered ethically "good" even from a human perspective to killswitch your own race? And "bad" from the moral standpoint of the planet-eating race?

The easiest way to dissolve such obvious contradictions is to say that there is just not, in fact, a universal hierarchy ranking ethical systems universally, regardless of the nature of the (rational = capable reasoner) agent.

Doesn't mean an agent isn't allowed to strongly defend what it considers to be moral, to die for it, even.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-25T16:05:58.276Z · LW(p) · GW(p)

Wouldn't that presuppose that "cooperation is the source/the sine qua non of all good"?

The point is it doesn't matter what you consider "good"; fighting people wont produce it (even if you value fighting people, because they will beat you and you'll be unable to fight.)

I'm not saying your goals should be ethical; I'm saying you should be ethical in order to achieve your goals.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-25T16:23:49.653Z · LW(p) · GW(p)

That seems very simplistic.

Ethically "good" = enabling cooperation, if you are not cooperating you must be "fighting"?

Those are evidently only rough approximations of social dynamics even just in a human context. Would it be good to cooperate with an invading army, or to cooperate with the resistance? The one with an opposing goal, so as a patriot, the opposing army it is, eh?

Is it good to cooperate with someone bullying you, or torturing you? What about game theory, if you're not "cooperating" (for your value of cooperating), you must be "fighting"? What do you mean by fighting, physical altercations? Is a loan negotiation more like cooperation or more like fighting, and is it thus ethically good or bad, for your notion of "ethics = ways for agents with different goals to co-operate"?

It seems like a nice soundbite, but doesn't make even cursory sense on further examination. I'm all for models that are as simple as possible, but no simpler. But cooperation as the definition of ethics? For you, maybe. Collaborateur!

Replies from: MugaSofer
comment by MugaSofer · 2013-04-26T11:53:04.486Z · LW(p) · GW(p)

Fighting in this context refers to anything analogous to defecting in a Prisoner's Dilemma. You hurt the other side but encourage them to defect in order to punish you. You should strive for the Pareto Optimimum.

Maybe this would be clearer if we talked in terms of Pebblesorters?

comment by PrawnOfFate · 2013-04-20T00:42:09.020Z · LW(p) · GW(p)

Ah, but Kawoomba doesn't expect ethics to regulate other people, because he thinks everyone has incompatible goals. Thus ethics serves purely to define your goals.

Why not just say there is no ethics? His theory is like saying that since teapots are made of chocolate, their purpose is to melt into a messy puddle instead of making tea.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-23T11:01:43.638Z · LW(p) · GW(p)

I'm all in favor of him just using the word "goals", myself, and leaving us non-paperclippers the word "ethics", but oh well. It confuses discussion no end, but I guess it makes him happy.

Also, arguing over the "correct" word is low-status, so I'd suggest you start calling them "normative guides" or something while Kawoomba can hear you if you don't want to rehash this conversation. And they can always hear you.

ALWAYS.

comment by PrawnOfFate · 2013-04-18T16:03:47.083Z · LW(p) · GW(p)

"There is a non-zero chance of one correct ethical system existing, as long as that's there, I'm free to believe it", or what?

If there is a system of objective morality based on reason, then I am rationally compelled to believe it.

No Sir, if you insist there is any basis whatsoever to stake your "one ethics to rule them all" claim on, you argue it's more likely than not.

My actual claim, for the third time, is that relativism is not obviously true and realism is not obviously false. That does not require "more likely to be right than wrong".

I do not stake my belief on absolute certainties, that's counter to all the tenets of rationality, Bayes, updating on evidence et al.

Neither do I. I never said anything of the kind. You keep trying to shoehorn what I am saying into your preconceived notion of Arguing With a Theist. Please don't.

My argument is clear. Different agents deem different courses of actions to be good or bad.

True but irrelevant. Doesn't prove relativism.

There is a basis (such as Aumann's) for rational agents to converge on isomorphic descriptions of the world. There is no known, or readily conceivable, basis for rational agents to all converge on the same course of action.

That is nothing but gainsaying my argument. I have sketched how rational agents could become persuaded of rationally based ethics as they are of any other rational proposition.

On the contrary, that would entail that e.g. world-eating AIs that are also smarter than any humans, individual or collectively, cannot possibly exist.

Yep. I think Clipper arguments are contestable.

There are no laws of physics preventing their existence - or construction [ of world eating AIs].

There are laws of logic preventing the conjunction of "is hyper rational" and "arbitrarily ignores rationally compelling clams".

So we should presume that they can exist. If their rational capability is greater than our own, we should try to adopt world eating, since they'd have the better claim (being smarter and all) on having the correct ethics, no?

Show me one, and I'll consider it. But why should I abandon my present beliefs just because a hyperratioanl agent might believe something else? A hyperrational agent might believe anything else. It cancels out. A la Pascal's Wager.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-19T12:46:31.745Z · LW(p) · GW(p)

Neither do I. I never said anything of the kind. You keep trying to shoehorn what I am saying into your preconceived notion of Arguing With a Theist. Please don't.

Upvoted just for this, and also for responding civilly and persuasively to Kawoomba's ... Kawoombaing.

Also, I think you might like this relevant link. I know I did.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-19T16:15:41.054Z · LW(p) · GW(p)

Also, I think you might like this relevant link.

I did. But I was a bit puzzled by this,..

" we should believe the Bible because the Bible is correct about many things that can be proven independently, this vouches for the veracity of the whole book, and therefore we should believe it even when it can't be independently proven"

.. which,even as the improved version of, a straw man argument is still pretty weak. The Bible is a compendium of short books written by a number of people at disparate periods of time. The argument would work much better about a more cohesive work, such as the Koran....

(No, Kawoomba, I did not admit to being a Muslim...)

Replies from: MugaSofer
comment by MugaSofer · 2013-04-19T17:29:46.962Z · LW(p) · GW(p)

Well, it's not an argument I'd personally make in this case (for roughly the reasons you outline) but it's not an argument that's trivially wrong from the outset; you have to actually engage in biblical scholarship to understand the flaw.

And at least it's not circular.

comment by PrawnOfFate · 2013-04-18T14:31:48.321Z · LW(p) · GW(p)

So is it a component of the "correct" ethical preferences that they satisfy the preferences of others?

Take into account, at least. In which case: of course. An "ethics" that was all about your own preferences would be vacuous--it would just be a duplicate of instrumental rationality.

comment by PrawnOfFate · 2013-04-18T14:47:08.413Z · LW(p) · GW(p)

But hold on, our ethical preferences aren't designed to maximize other sapients' preferences. Wouldn't it be more ethical still to not want anything for yourself

Not necessarily. Ethics uncontentiously includes fairness. Treating an arbitrary person's preferences as being unimportant would be unfair, so treating your own preferences as unimportant would be unfair.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-18T14:48:48.792Z · LW(p) · GW(p)

No, no. Wouldn't it be more ethical if your preferences were "I want nothing above strict subsistence".

You can take those preferences as seriously and important as anything.

More ethical, no?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-18T14:53:48.928Z · LW(p) · GW(p)

Wouldn't it be more ethical if your preferences were "I want nothing above strict subsistence".

That would still be unfair if you want more than strict subsistence for others.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-18T14:57:31.824Z · LW(p) · GW(p)

That would still be unfair if you want more than strict subsistence for others.

Why would you? Wouldn't a society in which everyone's preference would be to want nothing above strict subsistence be maximally satisfied if they all had nothing above strict subsistence?

They can take that very seriously, and all be maximally ethical, much more so than us. Huzzah!

(If we fire off comments like that, let's consolidate the different lines of comments into one.)

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-18T15:01:49.834Z · LW(p) · GW(p)

Why would you? Wouldn't a society in which everyone's preference would be to want nothing above strict subsistence be maximally satisfied if they all had nothing above strict subsistence?

Whatever. People aren;t actually like that. What is your point?

They can take that very seriously, and all be maximally ethical, much more so than us. Huzzah!

But we are not suddenly going to stop wanting what we want. What is your point?

Replies from: Kawoomba
comment by Kawoomba · 2013-04-18T15:07:28.370Z · LW(p) · GW(p)

What is your point?

That your supposedly objectively-ethically-correct-for-all-minds "must maximize everyone's preferences, including my own" ethics would score some strange society as the one I've outlined higher than anything humans could achieve. So that's what your own correct ethics tell you to aspire to, no?

It's a reductio ad absurdum, what else?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-18T15:16:27.738Z · LW(p) · GW(p)

I dont think such a society is more virtuous, it is just a society where the bar is lower. The flipside is resource-rich societies where it is easier to do the right thing because there are more resources. That isnt more virtuous either, because it is not vicious to be unable to do the right thing because of lack or resources. Virtue and vice are about intention..

Replies from: Kawoomba
comment by Kawoomba · 2013-04-18T15:21:35.242Z · LW(p) · GW(p)

(Without knowing, I'm guessing you're a Christian at 5:1, that you're a theist at 10:1)

I dont think such a society is more virtuous, it is just a society where the bar is lower. The flipside is resource-rich societies where it is easier to do the right thing because there are more resources.

On the contrary, using less resources to satisfy yourself and others, all the other resources would be free to create more fully satisfied and happy beings. If you're saving a lot of money and not buying yourself gadgets, that increases your ability to effect change, not diminishes it.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-18T15:31:55.499Z · LW(p) · GW(p)

(Without knowing, I'm guessing you're a Christian at 5:1, that you're a theist at 10:1)

Why? My approach is explictly non-euthyphric. However, I notice you keep arguing with me as though I am a theist..

If you're saving a lot of money and not buying yourself gadgets, that increases your ability to effect change, not diminishes it.

I am trying to distinguish between two sides of morlality--doing the right thing, and Virtue (AKA wanting to do the right thing).

Replies from: MugaSofer
comment by MugaSofer · 2013-04-19T12:34:39.558Z · LW(p) · GW(p)

Quick, quick! Make a bet that his stereotypical assumption is wrong!

It, um, is wrong, right?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-19T12:45:56.427Z · LW(p) · GW(p)

The funny thing is that if one participant in a discussion makes clear statements, and the other reads them carefully, there isn't the slightest need for that kind of guesswork.

comment by TheOtherDave · 2013-04-18T13:12:17.606Z · LW(p) · GW(p)

Can you expand on how you got the "preferences are the same" part?

Replies from: ArisKatsaris
comment by ArisKatsaris · 2013-04-18T13:20:34.909Z · LW(p) · GW(p)

I thought we were keeping everything else the same, and reversing only the ethics.

In a world where everyone preferred to be murdered as soon as possible, I can agree that murder may very well be ethical.

Replies from: TheOtherDave
comment by TheOtherDave · 2013-04-18T15:02:32.848Z · LW(p) · GW(p)

What do you want to say about a world where everyone agreed that there were some people who they preferred be murdered, and some people they preferred not be murdered, and that it's ethical to murder people you prefer to be murdered, even if everyone doesn't necessarily agree on which people fall into which category?

comment by Estarlio · 2013-04-18T13:13:53.365Z · LW(p) · GW(p)

It would be correctly describing its preferences, and its preferences would not be ethically correct. You could construct an AI that frimly believed 2+2=5. And it would be wrong. As before, you are glibly assuming that the word "ethical" does no work, and can be dropped from the phrase "ethical value".

Well, what work does it do? You haven't pointed to or defined ethically it's difficult to see how your statement is expected to parse:

"Their values wouldn't be [untranslatable 1] correct." is more or less what I'm getting at the moment.

What are you actually talking about? Where's your information for this idea that some values are 1+1=3 style incorrect coming from?

Replies from: MugaSofer, PrawnOfFate
comment by MugaSofer · 2013-04-19T12:19:20.131Z · LW(p) · GW(p)

It's worth noting that they would definitely be "unethical" if we define "ethical" in terms of our own preferences. It's a rigid designator, just not one inscribed on a stone tablet at the center of the universe.

comment by PrawnOfFate · 2013-04-18T15:11:23.980Z · LW(p) · GW(p)

I didn't define any of the other words I used either. "Ethics" isn't a word I invented.

Where's your information for this idea that some values are 1+1=3 style incorrect coming from?

Moral realism. Shelves full of books have been written about it over many centuries. Why has no-one here heard of it?

Replies from: Estarlio, MugaSofer
comment by Estarlio · 2013-04-18T16:30:15.834Z · LW(p) · GW(p)

Moral realism. Shelves full of books have been written about it over many centuries. Why has no-one here heard of it?

Moral realism has been formulated in a great number of ways over the years. In my opinion never convincingly. A guy further up the thread mentioned the form of it you seem to be using.

Perhaps I was unclear. Where is your second correlate? What are you mapping onto? Where's your information coming from that you're right or wrong in light of?

If you just mean something to the effect of one should always act in a way that favours one's most dominant long-term interests, that seems to be the typical situational pragmatism account of normative ethics. As such:

A) A matter of pragmatism rather than what people would generally mean by ethics. To roughly paraphrase some guy whose name I can't remember, 'As soon as they can get away with doing otherwise they become justified in doing so.'

&

B) Massively unactionable for most people. It's not clear that my higher order goals always outweigh a combination of lower order goals, or even that they should considering that rewards are going to vary over time.

I suppose you might formulate the idea that one should always act in the present such that one will have cause for the least regret in the future. That you would choose the same course of action for your past self looking back from the future as you would for your future self looking forwards from the past. Ethics would in other words be anti-akrasia.

And fair enough, maybe so. But now relating that back to discussion that you responded to I don't see how it serves one way or the other with respect to homosexuality and religion as preference choices, nor how it serves as a response to a refutation of moral universalism that arose in that discussion which you seemed to be replying to.

So - is that actually what you mean; how do you resolve the issues of relative weighting of preferences and changing situations; and if you resolve that, how do you apply it to the case in hand?

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-18T16:47:19.854Z · LW(p) · GW(p)

Where's your information coming from that you're right or wrong in light of?

The functional role of ethics places constraints on metaethical axioms or maxims, which, when combined with facts about preferences, can be concretised into an object level ethics.

So - is that actually what you mean; how do you resolve the issues of relative weighting of preferences and changing situations; and if you resolve that, how do you apply it to the case in hand?

I don't have a know what the One True Ethics is. I don't know what the One True Physics is either. That doesn't refute physical realism. The former doesn't refute metaethical realism. I am only arguing that realism is not obviously false, not relativism obviously true.

comment by MugaSofer · 2013-04-19T12:19:40.857Z · LW(p) · GW(p)

I think he wanted you to taboo it, dear.

Moral realism. Shelves full of books have been written about it over many centuries. Why has no-one here heard of it?

We mostly think it's disproved, to local standards of disproof.

Replies from: PrawnOfFate
comment by PrawnOfFate · 2013-04-19T12:34:35.195Z · LW(p) · GW(p)

I think he wanted you to taboo it, dear.

If you read his reply, he wanted me to explain what the truth-makers of moral propositions are.

I think he wanted you to taboo it, dear.

I noticed, sweetie. But it's belief in belief. If you have disproved something, you can repeat or cite the disproof. I have argued this topic out with at least half a dozen LWers, and none of them could put up a coherent case. Kawoomba gave up out, after apparently downvoting a bunch of my postings as a parting shot. That's the quality of argument.

comment by MugaSofer · 2013-04-19T12:17:22.249Z · LW(p) · GW(p)

It's a real position, if one based on rather questionable arguments.

OTOH, there really are some "values" that (sufficiently advanced) consequentialists will hold unless they specifically value not doing them, for instrumental reasons.

comment by MugaSofer · 2013-04-12T11:19:35.929Z · LW(p) · GW(p)

Welcome to LessWrong!

I'm playing catch-up, trying to expand my mind as fast as I can to make up for the lost years I spent blinded by religious dogma. Just two years ago, for example, I believed homosexuality was an evil that threatened to destroy civilization, that humans came from another planet, and that the Lost Ten Tribes were living somewhere underground beneath the Arctic. Needless to say, my re-education process has been exhausting.

Good for you! You might want to watch out for assuming that everyone had a similar experience with religion; many theists will fin this very annoying and this seems to be a common mistake among people with your background-type.

In trying to help him rediscover his faith, he had me read The God Delusion, which obliterated my own.

Huh. I must say, I found the GD pretty terrible (despite reading it multiple times to be sure,) although I suppose that powder-keg aspect probably accounts for most of your conversion (deconversion?)

I'm curious, could you expand on what you found so convincing in The God Delusion?

While I may not be a rationalist now, I would really like to be.

I think we can all say that :)

Replies from: atomliner, Kawoomba
comment by atomliner · 2013-04-14T10:07:34.596Z · LW(p) · GW(p)

Welcome to LessWrong!

Thank you! :)

Good for you! You might want to watch out for assuming that everyone had a similar experience with religion; many theists will fin this very annoying and this seems to be a common mistake among people with your background-type.

I apologize. I had no idea I was making this false assumption, but I was. I'm embarrassed.

I'm curious, could you expand on what you found so convincing in The God Delusion?

I replied to JohnH about this. I don't know if I could go into a lot of detail on why it was convincing, it was almost two years ago that I read it. But what really convinced me to start doubting my religion was when I prayed to God very passionately asking him whether or not The God Delusion was true and after I felt this tingly warm sensation telling me it was. I had done the same thing with The Book of Mormon multiple times and felt this same sensation, and I was told in church that this was the Holy Spirit telling me that it was true. I had been taught I could pray about anything and the Spirit would tell me whether or not it was true. After being told by the Spirit that The God Delusion was true, I decided that the only explanation is that what I thought of as the Spirit was just happening in my head and that it wasn't a sure way of finding knowledge. It was a very dramatic experience for me.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-14T14:49:30.313Z · LW(p) · GW(p)

I prayed to God very passionately asking him whether or not The God Delusion was true and after I felt this tingly warm sensation telling me it was. I had done the same thing with The Book of Mormon multiple times and felt this same sensation, and I was told in church that this was the Holy Spirit telling me that it was true. I had been taught I could pray about anything and the Spirit would tell me whether or not it was true. After being told by the Spirit that The God Delusion was true, I decided that the only explanation is that what I thought of as the Spirit was just happening in my head and that it wasn't a sure way of finding knowledge.

I've always wondered about that. People talk about praying for guidance and receiving it, never quite got what they were talking about before now.

Yeah, I suppose what you describe fits with it being more that the book encouraged you to reexamine your beliefs than it's arguments persuading you as such, which makes sense.

Incidentally, I can't help wondering what would you have done if the Spirit had told you it was bunk ;)

Replies from: atomliner
comment by atomliner · 2013-04-14T21:08:09.424Z · LW(p) · GW(p)

Incidentally, I can't help wondering what would you have done if the Spirit had told you it was bunk ;)

I like to think I still would have debunked Mormonism in my own mind, but maybe not! That experience was extremely important to my deconversion process, because the only reason I believed in the LDS Church was because of the Spirit telling me the Book of Mormon was true and that Jesus Christ was my Savior. As soon as the Spirit told me something so contradictory as The God Delusion was true, my whole belief structure came crumbling down.

comment by Kawoomba · 2013-04-12T11:28:51.555Z · LW(p) · GW(p)

What kind of theist are you, personal or more of the general theism (which includes deism) variety? Any holy textstring you believe has been divinely inspired?

Replies from: MugaSofer
comment by MugaSofer · 2013-04-12T17:19:53.306Z · LW(p) · GW(p)

About as Deist as you can be while still being technically Christian. I'd be inclined to say there's something in all major religions, simply for selection reasons, but the only thing I'd endorse as "divinely inspired" as such would be the New Testament? I guess? Even that is filtered by cultural context and such, obviously,.

Replies from: TheOtherDave
comment by TheOtherDave · 2013-04-12T19:43:52.840Z · LW(p) · GW(p)

If you can readily articulate your reasons for evaluating the New Testament differently from other scriptures, I'm interested. (It's possible that you've already done so, perhaps even in response to this question from me; feel free to point me at writeups elsewhere if you wish.)

Replies from: MugaSofer
comment by MugaSofer · 2013-04-12T20:12:15.672Z · LW(p) · GW(p)

Well, I mentioned I'm technically Christian (despite my deist leanings), right? I think the evidence in favor of Jesus being, well, the Son of God is good enough to over come my prior, although to be fair I have a significantly higher prior of such things than the LW norm, because theism. If Jesus was God, naturally anything that can be traced back to him is in some sense "divinely inspired" - so the Gospels, mostly. I'm less confident about the status of the rest of the NT, but again, probably miracles (albeit lower certainty than those of Jesus, I guess) so probably some level of Godly origin, at least for the parts that claim to have such an origin.

Replies from: TheOtherDave
comment by TheOtherDave · 2013-04-12T20:45:16.036Z · LW(p) · GW(p)

(nods) That answers my question. Thank you.

comment by Kawoomba · 2013-04-12T08:48:35.785Z · LW(p) · GW(p)

How many of your younger Mormon peers and friends do you think are secretly atheists?

Replies from: atomliner
comment by atomliner · 2013-04-12T09:43:29.893Z · LW(p) · GW(p)

I've only had two of my Mormon peers/friends/relatives reveal to me after knowing them for a substantial amount of time that they are atheists. Based on that, I would guess the percentage of active Latter-day Saints that are closet atheists is pretty low, around 1%-3%?

Replies from: CCC
comment by CCC · 2013-04-13T21:09:20.365Z · LW(p) · GW(p)

That implies that you have more-or-less a hundred close friends/peers/relatives, who you have known for a substantial amount of time and would expect them to tell you if they are closet atheists.

Replies from: Eliezer_Yudkowsky, atomliner
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-04-13T21:26:07.734Z · LW(p) · GW(p)

Mormons have lots of friends, and lots of relatives.

Replies from: Kawoomba
comment by Kawoomba · 2013-04-13T21:31:17.116Z · LW(p) · GW(p)

Mormons. M-o-r-m-o-n-s. The 'm' is silent.

Edit: Downvoted for something the majority probably agrees with, because I haven't wrapped it up in condescending niceties? We're talking about the Book of Mormon and following a charlatan like Joseph Smith. If that's still too ambiguous to render an opinion, what isn't?

Edit2: Becoming a rationalist and believing in some ancient religious scrolls are mutually exclusive. Dear reader (in most cases), you know this, I know this. People have seen the arguments presented a hundred times. Some of them still choose to believe in nonsense. At what point are we allowed to point out the stupidity of that belief? At what point can we stop politely rehashing arguments, and say "look, you're smart yet you abuse your smarts to rationalize nonsense".

That's the most dangerous kind of person, not some harmless peasant who can't know any better. A university educated student who can take flying lessons. A nuclear scientist who believes in some fundamentalist islamic regime. The smarter you are, the less excuses you have. What's the alternative, saying "well, I presented my arguments, you say your belief is perfectly rational. Let's leave it at that"?

Replies from: ArisKatsaris, orthonormal
comment by ArisKatsaris · 2013-04-15T10:52:17.255Z · LW(p) · GW(p)

because I haven't wrapped it up in condescending niceties?

Being nice is important.

If that's still too ambiguous to render an opinion, what isn't?

Kindergarten level insults like "Mormon sort-of-rhymes with Moron" aren't just an expression of opinion. Mormon would be sort-of-rhyming with Moron, even if Mormonism had been true. What you instead expressed is a cutesy and juvenile way of insulting someone: "The mormon is a moron, the mormon is a moron, hahahaha!"

comment by orthonormal · 2013-04-20T04:26:22.899Z · LW(p) · GW(p)

I downvoted your comment, not because I don't enjoy a good takedown of sloppy religious reasoning (even in the form of a snappy comment), but because this was nothing of the sort: it's a completely boring instance of saying "Boo X", without any content or even cleverness. It's just noise within the discussion.

Seriously, you know you can do better than that.

comment by atomliner · 2013-04-14T08:51:59.666Z · LW(p) · GW(p)

Over twenty-three years the numbers add up. I think I could easily find more than a hundred active Latter-day Saints just counting members of my extended family that I routinely encounter every year.

comment by JohnH · 2013-04-12T18:40:03.904Z · LW(p) · GW(p)

I am Mormon so I am curious where you got the beliefs that Homosexuality would destroy civilization, that humans came from another planet, that the Ten Tribes live underground beneath the Arctic? Those are not standard beliefs of Mormons (see for instance the LDS churches Mormonsandgays.org) and only one of those have I ever even encountered before (Ten Tribes beneath the Arctic) but I couldn't figure out where that belief comes from or why anyone would feel the need to believe it.

I also have to ask, the same as MugaSofer, could you explain how The God Delusion obliterated your faith? It seemed largely irrelevent to me.

Replies from: atomliner, atomliner
comment by atomliner · 2013-04-14T09:39:18.016Z · LW(p) · GW(p)

I have visited mormonsandgays.org. That came out very recently. It seems that the LDS Church is now backing off of their crusade against homosexuality and same-sex marriage. In the middle of the last decade, though, I can assure you what I was taught in church and in my family was that civilizations owed their stability to the prevalence of traditional marriages. I was told that Sodom and Gomorrah were destroyed because homosexuality was not being penalized and because of the same crime the Roman Empire collapsed. It is possible that these teachings, while not official doctrine, were inspired by the last two paragraphs of the LDS Church's 1995 proclamation The Family. In the second to last paragraph it says:

... we warn that the disintegration of the family will bring upon individuals, communities, and nations the calamities foretold by ancient and modern prophets. LINK

I have a strong feeling my interpretation of this doctrine is also held by most active believing American Mormons, having lived among them my entire life.

I don't think that most Mormons believe that mankind came from another planet, but I started believing this after I read something from the Journal of Discourses, in which Brigham Young stated:

Here let me state to all philosophers of every class upon the earth, When you tell me that father Adam was made as we make adobies from the earth, you tell me what I deem an idle tale. When you tell me that the beasts of the field were produced in that manner, you are speaking idle worlds devoid of meaning. There is no such thing in all the eternities where the Gods dwell. Mankind are here because they are the offspring of parents who were first brought here from another planet, and power was given them to propagate their species, and they were commanded to multiply and replenish the earth. LINK

This doctrine has for good reason been de-emphasized by the LDS Church, but never repudiated. I read this and other statements made by Brigham Young and believed it. I did believe he was a prophet of God, after all.

I began to believe that the Ten Tribes were living underneath the Arctic after reading The Final Countdown by Clay McConkie which details the signs that will precede the Second Coming. In the survey he apparently conducted of active Latter-day Saints, around 15% believed the Ten Tribes were living somewhere underground in the north. This belief is apparently drawn from an interpretation of Doctrine & Covenants 133:26-27, which states:

26 And they who are in the north countries shall come in remembrance before the Lord; and their prophets shall hear his voice, and shall no longer stay themselves; and they shall smite the rocks, and the ice shall flow down at their presence. 27 And an highway shall be cast up in the midst of the great deep.

I liked the interpretation that this meant there was a subterranean civilization of Israelites and believed it was true.

I apologize that I gave examples of these extraordinary former beliefs right after I wrote "I'm playing catch-up, trying to expand my mind as fast as I can to make up for the lost years I spent blinded by religious dogma." That definitely implies that these former beliefs were actual official doctrine of the Mormon Church. I did not intend that.

Replies from: JohnH
comment by JohnH · 2013-04-14T14:52:24.382Z · LW(p) · GW(p)

I am going to make a prediction that you likely grew up in a smaller community in Utah or Eastern Idaho.

In regards to the Journal of Discourse quote, the actual doctrine that Brigham Young is talking about it is very much emphasized and is found in the D&C, the Book of Abraham, and explicitly in the Temple. A dead giveaway is his reference to philosophers, he isn't talking about us being aliens but that our spirits always existed and come from where God is rather then being created at birth as thought in the rest of Christianity. Given this and your explanation of The God Delusion I take it you aren't that familiar with non-LDS Christian philosophy and the vast differences between us and them.

The church has not changed at all its position on same-sex marriage and just filed an amicus brief on the subject. I can see how your conclusion on the subject was drawn though.

Replies from: atomliner
comment by atomliner · 2013-04-14T21:33:37.424Z · LW(p) · GW(p)

I am going to make a prediction that you likely grew up in a smaller community in Utah or Eastern Idaho.

Wrong. I moved to Utah already an atheist. I didn't grow up in any one area, my family moved several times when I was younger. For example, I lived in Arizona, California, Georgia, and North Carolina before moving to Utah. The state I feel most confident in calling my home is California, since I lived there from 2004 to 2009.

In regards to the Journal of Discourse quote, the actual doctrine that Brigham Young is talking about it is very much emphasized and is found in the D&C, the Book of Abraham, and explicitly in the Temple. A dead giveaway is his reference to philosophers, he isn't talking about us being aliens but that our spirits always existed and come from where God is rather then being created at birth as thought in the rest of Christianity.

I highly disagree that this is what Brigham Young actually intended to teach. For example, in another part of the Journal of Discourses he says:

Though we have it in history that our father Adam was made of the dust of this earth, and that he knew nothing about his God previous to being made here, yet it is not so; and when we learn the truth we shall see and understand that he helped to make this world, and was the chief manager in that operation. He was the person who brought the animals and the seeds from other planets to this world, and brought a wife with him and stayed here. You may read and believe what you please as to what is found written in the Bible. Adam was made from the dust of an earth, but not from the dust of this earth. He was made as you and I are made, and no person was ever made upon any other principle. LINK

It does not seem at all that he is talking about the creation of their spirits, but the creation of their bodies.

Given this and your explanation of The God Delusion I take it you aren't that familiar with non-LDS Christian philosophy and the vast differences between us and them.

I have to admit I didn't regard myself as extremely familiar with Christian philosophy before my de-conversion, but I've learned a great deal since coming home from my mission. However, I don't think this was a very fair assessment of my knowledge on your part. There is nothing I've written that gives strong evidence for me being ignorant of Christian and Mormon theology. It seems to me you want to de-legitimize what I have to say by painting me as unintelligent and inexperienced with my own religion. Now, do you really think a person who has studied the Journal of Discourses wouldn't also most likely be a person who has spent a lot of time and energy investigating the rest of Mormon theology? I mean, a scholar I am not but I definitely know my way around Mormonism, more than most Mormons I know at the very least.

The church has not changed at all its position on same-sex marriage and just filed an amicus brief on the subject. I can see how your conclusion on the subject was drawn though.

There is a big difference between an "official position" and what is taught in the chapel and at the dinner table.

Replies from: JohnH
comment by JohnH · 2013-04-15T03:40:15.457Z · LW(p) · GW(p)

Since it appears that you grew up in a pluralistic society then I have no idea why you considered everyone different then you to not be a good person and feel you were never exposed to the idea that they possibly could be a good person. Considering that Jesus (Matthew 25:40), Paul (Romans 2), Nephi, Benjamin, Alma, and Moroni all say that it is action more then belief that defines who is saved, who has faith, and who is good, happy and healthy then I don't know how it was a shocking revelation that those who do not have the law but that act by nature according to law are just as much blessed as those that have the law.

I fail to see how blood atonement, Adam-God, racist theology, and polygamist theology gave you the slightest impression that the Journal of Discourses was a good source of doctrine. It is my personal experience that generally those that spend the most time reading it are those least familiar with the gospel, on either end of the spectrum. The biggest fans of the Journal of Discourses seem to be those that are trying to prove the church wrong and those that are seeking "deep" doctrine while ignoring the weightier parts of the gospel, by which I mean those that try to square Adam-God statements or that speculate on the location of the ten tribes or Kolob.

For instance nearly everyone that has taken the time to figure out what Christians say of God in their arguments for God and what the D&C says on the subject quickly realize that the two are wholly incompatible. That those beliefs on God and the arguments in favor of those beliefs are mixing Greek philosophy with scripture to synthesis a new belief. Not that members of the church are not also guilty of mingling the philosophies of men with scripture, that is a very common occurrence as you note with "what is taught in the chapel and the dinner table", me, I tend to focus on the current authorized messengers from God and the Holy Spirit as I feel that is what I have been instructed to do.

Replies from: atomliner, None, MugaSofer
comment by atomliner · 2013-04-15T04:36:12.814Z · LW(p) · GW(p)

I never said that I considered people different than me to not be good. What I said in earlier comments is that I liked The God Delusion because it introduced me to the concept that you can be "a good, healthy, happy person without believing in God". I believed that those who did not have faith in God would be more likely to be immoral, would be more likely to be unhealthy, and would definitely be more unhappy than if they did believe in God. The book presented to me a case for how atheists can be just as moral, just as healthy, just as happy as theists, an argument I had never seen articulated before. I apologize that I had never conjured this idea up before reading The God Delusion, it just seemed obvious to me based on my study of the Gospel that they couldn't be.

What passages in the scriptures tell you that you can be moral, healthy, and happy without faith in God? It seems pretty consistent to me that in the scriptures they say you can only have those qualities in your life if you believe in God and follow his commandments.

I fail to see how blood atonement, Adam-God, racist theology, and polygamist theology gave you the slightest impression that the Journal of Discourses was a good source of doctrine.

I believed in blood atonement, the Adam-God theory, much of the racist theology, and in polygamy. Why wouldn't I? The prophets speak for God. God would not let a prophet lead the Church astray. My patriarchal blessing told me to always follow the prophets. No one ever told me I could question and disagree with the prophets and still be a member of the LDS Church in good standing. I apologize that I didn't come to the same understanding a you, but I don't see any reason I would have with the life experiences I had.

The biggest fans of the Journal of Discourses seem to be those that are trying to prove the church wrong and those that are seeking "deep" doctrine while ignoring the weightier parts of the gospel, by which I mean those that try to square Adam-God statements or that speculate on the location of the ten tribes or Kolob.

I really liked "deep doctrine". :) If I had been alive during the Roman Empire I think I would have been a sucker for the mystery cults. Still, what do you even mean by the "weightier parts of the gospel"? I feel like that is so subjective as to be meaningless in our conversation. How can we determine objectively which parts of the Gospel is more important?

I tend to focus on the current authorized messengers from God and the Holy Spirit as I feel that is what I have been instructed to do.

Great. I have no problem with you finding a way to make it all work in your head. Obviously I couldn't discover how to make it work in mine after discovering things that I did. No amount of instruction could keep it all from unraveling.

I wish you weren't so hostile against me just because I'm making your in-group look bad.

Replies from: JohnH
comment by JohnH · 2013-04-15T14:37:51.739Z · LW(p) · GW(p)

What passages in the scriptures tell you that you can be moral, healthy, and happy without faith in God?

I already pointed you to Romans 2, specifically in this case Romans 2:13-15, did you want more?

Why wouldn't I?

A prophet is only a prophet when they are acting as a prophet. More specifically there are multiple First Presidency statements saying Adam-God is wrong; Statements by Apostles saying that the racist theology was created with limited understanding and is wrong (as well as more recent church statements saying explicitly that it is contrary to the teachings of Christ); I am not referring to polygamy as a practice but the belief that polygamy is the new and everlasting covenant itself, which again has revelation and first presidency statements and even the scriptures on polygamy saying that is wrong; Also given that none of those theories were presented to the Quorums of the Church and that Apostles and a member of the First Presidency disagreed vocally with Adam-God at the time I would have thought it was clear that one can disagree with ideas not presented as revelation and not sanctioned by the First Presidency and Quorum of the Twelve. I mean, the D&C has procedures on how to conduct a disciplinary council of the prophet so while the prophet will not lead the church astray they are quite capable of sinning and of theorizing based of revelation and their own prejudices as anyone else, though they seem to have mostly gotten better at not doing that.

what do you even mean by the "weightier parts of the gospel"?

The two great commandments: Love God, Love your neighbor as yourself, and the actual gospel: faith, repentance, baptism, and the gift of the Holy Ghost. Statements of the effect of that being sufficient for anyone or even that being the doctrine of Christ and the only doctrine of Christ and anything more or less being declared as the doctrine of Christ being evil seem fairly objective in stating which parts of the Gospel are most important.

Replies from: atomliner
comment by atomliner · 2013-04-15T16:28:00.880Z · LW(p) · GW(p)

I already pointed you to Romans 2, specifically in this case Romans 2:13-15, did you want more?

Yes. I don't see anything in Romans 2 that shows me that you can be moral, healthy, and happy without faith in God.

A prophet is only a prophet when they are acting as a prophet.

But you have to admit it's hard sometimes to distinguish whether or not a prophet is acting as one.

More specifically there are multiple First Presidency statements saying Adam-God is wrong.

I never believed that Adam WAS Elohim, but I did believe that what Brigham Young and others intended to say was that Adam was the God of this Earth.

Statements by Apostles saying that the racist theology was created with limited understanding and is wrong

I never believed that black people were cursed for being fence-sitters in the War in Heaven, but I did believe that it was because of the curse of Cain that they couldn't have the priesthood until 1978. In my defense I started believing around 2009 that the priesthood ban was just an incorrect Church policy. Still, I never read anything from the Apostles saying that the priesthood ban was wrong, just that it was unknown why there was a priesthood ban.

I am not referring to polygamy as a practice but the belief that polygamy is the new and everlasting covenant itself, which again has revelation and first presidency statements and even the scriptures on polygamy saying that is wrong.

I always believed that the new and everlasting covenant was referring to celestial marriage, but I did believe that polygamy would eventually be re-instated being that before the Second Coming there would have to be a restitution of all things.

Also given that none of those theories were presented to the Quorums of the Church and that Apostles and a member of the First Presidency disagreed vocally with Adam-God at the time I would have thought it was clear that one can disagree with ideas not presented as revelation and not sanctioned by the First Presidency and Quorum of the Twelve.

I really only developed an understanding of Official Doctrine after my deconversion. Before, however, my understanding was that every member of the First Presidency and the Quorum of the Twelve Apostles were prophets, seers, and revelators and that they spoke directly with Jesus Christ, therefore they were incapable of teaching false doctrine to the members of the Church.

what do you even mean by the "weightier parts of the gospel"?

The two great commandments: Love God, Love your neighbor as yourself, and the actual gospel: faith, repentance, baptism, and the gift of the Holy Ghost.

You were saying how those who read the Journal of Discourses "seem to be those that are trying to prove the church wrong and those that are seeking 'deep' doctrine while ignoring the weightier parts of the gospel". I think you were trying to put me in the latter category, suggesting that I was ignoring what was really important in the Gospel. Now that you've explained what these "weightier parts" are, I assure you that I did not ignore these teachings. Those are incredibly simple and basic concepts that I had known for years and years. How could anyone ignore these parts of the Gospel while studying "deep doctrine"?

How long have you been a member of the LDS Church?

Replies from: MugaSofer, JohnH
comment by MugaSofer · 2013-04-17T12:19:18.861Z · LW(p) · GW(p)

I really only developed an understanding of Official Doctrine after my deconversion. Before, however, my understanding was that every member of the First Presidency and the Quorum of the Twelve Apostles were prophets, seers, and revelators and that they spoke directly with Jesus Christ, therefore they were incapable of teaching false doctrine to the members of the Church.

I wonder how common this is?

Replies from: atomliner
comment by atomliner · 2013-04-17T20:21:07.564Z · LW(p) · GW(p)

I masquerade as a liberal Mormon on Facebook since I'm still in the closet with my unbelief. In my discussions with friends and family the most common position taken is that the First Presidency and the Twelve Apostles cannot teach false doctrine or else they will be forcibly removed by God. I even had a former missionary companion tell me that President Gordon B. Hinckley died in 2008 not from old age (he was 98) but because he had made false statements on Larry King Live concerning the doctrine of exaltation in which worthy Latter-day Saints can become gods.

Replies from: Desrtopa, MugaSofer
comment by Desrtopa · 2013-04-17T20:26:21.799Z · LW(p) · GW(p)

How do they distinguish between true statements which precede their deaths, and false statements which cause their deaths?

Replies from: atomliner
comment by atomliner · 2013-04-17T20:55:30.495Z · LW(p) · GW(p)

Whatever the prophet says that doesn't match up with their own interpretation of Mormonism is false? I honestly do not know, I never thought this way when I was LDS.

comment by MugaSofer · 2013-04-19T13:15:05.866Z · LW(p) · GW(p)

Interesting.

comment by JohnH · 2013-04-15T17:11:35.518Z · LW(p) · GW(p)

you can be moral, healthy, and happy without faith in God

Paul saying those that didn't know God and that didn't have the law but that acted justly being justified because of their actions doesn't imply to you that it is possible to be moral, healthy, and happy without faith in God? How about this, where in "There is a law, irrevocably decreed in heaven before the foundations of this world, upon which all blessings are predicated— And when we obtain any blessing from God, it is by obedience to that law upon which it is predicated." does it mention anything about having faith in God being a prerequisite for receiving a blessing? Where in "if ye have done it unto the least of these they brethren ye have done it unto me" does it say that one must believe in God for that to be valid?

"Till you have learnt to serve men, how can you serve spirits?"

But you have to admit it's hard sometimes to distinguish whether or not a prophet is acting as one.

" Would God that all the LORD'S people were prophets, and that the LORD would put his spirit upon them!"

How could anyone ignore these parts of the Gospel while studying "deep doctrine"?

Very easily, as Jesus repeatedly stated.

every member of the First Presidency and the Quorum of the Twelve Apostles were prophets, seers, and revelators and that they spoke directly with Jesus Christ, therefore they were incapable of teaching false doctrine to the members of the Church.

I am not sure how the first part of this lead to the second part of this, but I will believe that was your belief.

How long have you been a member of the LDS Church?

My whole life.

Replies from: atomliner, MugaSofer
comment by atomliner · 2013-04-15T19:35:26.117Z · LW(p) · GW(p)

Paul saying those that didn't know God and that didn't have the law but that acted justly being justified because of their actions doesn't imply to you that it is possible to be moral, healthy, and happy without faith in God?

I don't know where you draw that implication from the word "justified". So, no.

How about this, where in "There is a law, irrevocably decreed in heaven before the foundations of this world, upon which all blessings are predicated— And when we obtain any blessing from God, it is by obedience to that law upon which it is predicated." does it mention anything about having faith in God being a prerequisite for receiving a blessing?

I guess I did have a very abstract belief that those who followed the commandments, the "law", even if they didn't believe, would still receive the same blessings as those who do. But, the part of The God Delusion that talked about atheists being just as happy, moral, and healthy as theists never said anything about following Mormon commandments to do so, and for that reason it was a revolutionary concept to me. What was a new concept was that you could have a lifestyle completely different from those lived by Latter-day Saints and still be moral, happy, and healthy. Though, come to think about it, I was introduced to this concept not just in The God Delusion, but also in my interactions with hundreds of Brazilian families. Certainly the mission experience added to the knowledge base I needed to refute Mormonism.

Where in "if ye have done it unto the least of these they brethren ye have done it unto me" does it say that one must believe in God for that to be valid?

"Till you have learnt to serve men, how can you serve spirits?"

" Would God that all the LORD'S people were prophets, and that the LORD would put his spirit upon them!"

The reason we are having this discussion is because I feel you've characterized me unfairly as "the Ex-Mormon who never really knew his own religion and had no reason to believe in the fringe theories he did". My goal is to support my case that I really was a mainstream Latter-day Saint before I lost my faith. So, you can use your apologetic arguments all you want for whatever idea you have about Mormonism, but if they aren't based clearly in the scriptures (which I studied a great deal), and if they were never taught widely in the Church, then why exactly did I err in not coming to the same understanding as you? I do not think you have any good evidence for why I was an atypical Mormon who was unjustified in believing in the things I did.

How could anyone ignore these parts of the Gospel while studying "deep doctrine"?

Very easily, as Jesus repeatedly stated.

What Jesus stated on this is extremely illogical to me. Why is what he said logical to you?

every member of the First Presidency and the Quorum of the Twelve Apostles were prophets, seers, and revelators and that they spoke directly with Jesus Christ, therefore they were incapable of teaching false doctrine to the members of the Church.

I am not sure how the first part of this lead to the second part of this, but I will believe that was your belief.

So you think that it was unreasonable for me to assume that men who are given an office BY GOD with the title "prophet, seer and revelator" and who speak directly with Jesus Christ, face-to-face, would not teach false doctrine? Do you think that a person who speaks face-to-face with Jesus Christ would then teach his own false ideas to members of Christ's One True Church?

My whole life.

And you are how old?

Replies from: CCC, MugaSofer, JohnH
comment by CCC · 2013-04-15T20:47:53.583Z · LW(p) · GW(p)

The reason we are having this discussion is because I feel you've characterized me unfairly as "the Ex-Mormon who never really knew his own religion and had no reason to believe in the fringe theories he did". My goal is to support my case that I really was a mainstream Latter-day Saint before I lost my faith.

So, this whole debate is about whether your-previous-self, or JohnH, is better deserving of the title of 'true Mormon'?


On a different point:

How could anyone ignore these parts of the Gospel while studying "deep doctrine"?

Very easily, as Jesus repeatedly stated.

What Jesus stated on this is extremely illogical to me. Why is what he said logical to you?

I would like to draw a figurative circle around this statement...

So you think that it was unreasonable for me to assume that men who are given an office BY GOD with the title "prophet, seer and revelator" and who speak directly with Jesus Christ, face-to-face, would not teach false doctrine? Do you think that a person who speaks face-to-face with Jesus Christ would then teach his own false ideas to members of Christ's One True Church?

...and compare it to this one. They appear to contradict each other. Can you explain?

Replies from: atomliner, JohnH
comment by atomliner · 2013-04-15T21:03:39.267Z · LW(p) · GW(p)

So, this whole debate is about whether your-previous-self, or JohnH, is better deserving of the title of 'true Mormon'?

That's funny. No. I don't care what JohnH wants to be seen as or what title he deserves. I just want my previous-self identified as a "plausible Mormon". In my opinion, JohnH wants me to be seen as a "fringe Mormon" whose departure from the LDS Church is unimportant in the debate over whether the LDS Church is true, because I didn't really understand Latter-day Saint beliefs. Which I did as much as any other average Latter-day Saint I know.

They appear to contradict each other. Can you explain?

I don't see the contradiction. These statements appear to be unrelated. Can you explain what contradiction you see?

Replies from: Estarlio, CCC, Bugmaster
comment by Estarlio · 2013-04-15T22:20:59.346Z · LW(p) · GW(p)

That's funny. No. I don't care what JohnH wants to be seen as or what title he deserves. I just want my previous-self identified as a "plausible Mormon". In my opinion, JohnH wants me to be seen as a "fringe Mormon" whose departure from the LDS Church is unimportant in the debate over whether the LDS Church is true, because I didn't really understand Latter-day Saint beliefs. Which I did as much as any other average Latter-day Saint I know.

If you're feeling trapped into arguing with this guy to defend your reputation, you may be better off just saying something like: "If you turn out to be right, and most people don't believe the way I do, I'm still not going to start believing in the LDS. Therefore my expected return on this conversation is 0 and I'm not going to continue it."

Certainly from my perspective that would be a much more high-status move than continuing to argue with the guy. Because, in all kindness: Your departure from the LDS is unimportant in the debate over whether the Church is true. Not because the beliefs are or are not commonly held, nor because they are or are not ridiculous, but because there are much better reasons for disbelieving. Whichever one of your views prevails here, it's not going to serve as a good reason for me or anyone else to start believing or disbelieving.

Your reasons may be important in a discussion over why people leave the LDS - but that's a separate issue to whether the LDS is true. So, you may not be getting what you think you're getting in terms of reputation by arguing this over this.

Replies from: atomliner
comment by atomliner · 2013-04-15T22:35:42.652Z · LW(p) · GW(p)

Those are strong arguments for discontinuing this discussion. Thank you for helping me grok this situation better. :)

comment by CCC · 2013-04-16T07:34:39.712Z · LW(p) · GW(p)

I don't see the contradiction. These statements appear to be unrelated. Can you explain what contradiction you see?

Well, let me start with the first example:

How could anyone ignore these parts of the Gospel while studying "deep doctrine"?

Very easily, as Jesus repeatedly stated.

What Jesus stated on this is extremely illogical to me. Why is what he said logical to you?

Paraphrasing somewhat, JohnH said 'because Jesus said so' and you responded that this reason was insufficient for a Mormon to hold a belief; that it needed to be logical as well.

While, in the second case...

So you think that it was unreasonable for me to assume that men who are given an office BY GOD with the title "prophet, seer and revelator" and who speak directly with Jesus Christ, face-to-face, would not teach false doctrine? Do you think that a person who speaks face-to-face with Jesus Christ would then teach his own false ideas to members of Christ's One True Church?

...it seems that you are claiming that saying that 'a man who has spoken with Jesus said so' is sufficient reason for a Mormon to hold a belief.

I would expect the second reason to be weaker than the first, since in the second case there is someone else speaking in the middle (if you've ever played Broken Telephone, you'll know why this is a bad thing). Yet you appear to be claiming that the second reason is stronger than the first. Hence my confusion.

Replies from: atomliner
comment by atomliner · 2013-04-16T21:21:19.186Z · LW(p) · GW(p)

Oh, okay, I understand how this could be seen as contradictory.

In the first case I was arguing from my own, real-time atheist self that believes Jesus was illogical in his comments on people forgetting the basic principles of Christianity in their pursuit for more knowledge. How could someone forget such simple principles like "love one another" in their pursuit for more knowledge? Note that I never said this reason was insufficient for a Mormon to hold this belief, I was only saying it was insufficient to atheist me and I wanted JohnH to provide a better defense of his point, which he didn't.

In the second case I used past-tense "... you think that it WAS unreasonable for me...", and we were already talking about my former beliefs. So, I was arguing from my former Mormon self that did believe that Jesus saying something was enough to validate a belief.

The discussion became rather confusing because JohnH wanted to discredit my past beliefs rather than my current beliefs.

Replies from: CCC
comment by CCC · 2013-04-17T08:36:34.189Z · LW(p) · GW(p)

Ah, I see. You were trying to defend two contradictory positions, and I did not notice when you switched between them. (This is one reason why I find it's often a bad idea to try to defend an idea that you have abandoned, by the way; it leads to confusion.)

How could someone forget such simple principles like "love one another" in their pursuit for more knowledge?

That is actually quite possible. Step one is a person who seeks more knowledge, and finds it. That's fine, so far. Step two is the person realises that they are a lot more knowledgeable than anyone else; that's fine as well, but it can be like standing on the edge of a cliff. Step three is that the person becomes arrogant. They see most other people as a distraction, as sort of sub-human. This is where things start to go wrong. Step four is when the person decides that he knows what the best thing for everyone else to do is better than they do. And if they won't do it, then he'll make them do it.

Before long, you could very well have a person who, while he admits that it's important to love your fellow-man in theory, in practice thinks that the best thing to do is to start the Spanish Inquisition. The fact that the Spanish Inquisition ever existed, started by people who professed "love one another" as a core tenet of their faith, shows that this can happen...

Replies from: atomliner, MugaSofer
comment by atomliner · 2013-04-17T20:13:26.312Z · LW(p) · GW(p)

Those are good examples. Though I guess whether this is possible depends on your definition of "forget". Speaking of the Spanish Inquisition, I am of the opinion that the Inquisitors did not forget their core tenets but that further knowledge (however flawed) gave them new means to interpret the original tenets. You could suggest that this re-interpretation was exactly what Jesus wanted to keep people from doing, of course. The question I ask Christians, then, is "What knowledge is acceptable and how should it be attained when God doesn't encourage the utilization of all knowledge?" This would certainly be an important question for theists to answer, and may be relatively simple. I can already guess a few possible answers.

Replies from: CCC
comment by CCC · 2013-04-17T20:50:35.141Z · LW(p) · GW(p)

Though I guess whether this is possible depends on your definition of "forget".

I'm assuming "to act as though ignorant of the principle in question".

The question I ask Christians, then, is "What knowledge is acceptable and how should it be attained when God doesn't encourage the utilization of all knowledge?"

I don't think its the knowledge that's dangerous, in itself. I think it's the arrogance. Or the sophisticated argument that starts with principles X and Y and leads to actions that directly contradict principle X.

For example; consider the following principles:

  1. Love thy neighbour as thyself
  2. Anyone who does not profess will be tortured terribly in Hell after death, beyond anything mortals can do

That's enough to lead to the Inquisition, by this route:

Looking at Principle 2, I do not wish myself, or those that I love to enter Hell. Considering Principle 1, I must try to save everyone from that fate, by any means possible. I must therefore attempt to convert everyone to .

(Consideration of various means snipped for brevity)

Yet there may be some people who refuse to convert, even in the face of all these arguments. In such a case, would torture be acceptable? If a person who is not tortured does not repent, then he is doomed to what is worse than a mere few months, even a mere few years of torture; he is doomed to an eternity of torture. If a person is tortured into repentance, then he is saved an eternity of torture - a net gain for the victim. If he is tortured and does not repent, then he experiences an eternity of torture in any case - in that case, he is at least no worse off. So a tortured victim is at worst no worse off, and at best a good deal better off, than a man who does not repent. However, care must be taken to ensure that the victim does not die during torture, but before repenting.

Better yet, the mere rumour of torture may lead some to repent more swiftly. Thus, judicious use of torture becomes a moral imperative.

(As an exercise, incidentally, can you spot the flaw in that chain of reasoning?)

And then you have the Inquisitors, and fear and terror and sharp knives in dark rooms...

comment by MugaSofer · 2013-04-17T11:05:55.621Z · LW(p) · GW(p)

Step four is when the person decides that he knows what the best thing for everyone else to do is better than they do. And if they won't do it, then he'll make them do it.

It's worth noting that if the person successfully "found knowledge", they are in fact correct (unless it was irrelevant knowledge, I guess.)

Replies from: CCC, RichardKennaway
comment by CCC · 2013-04-17T19:18:48.701Z · LW(p) · GW(p)

Historical evidence suggests that people get to step 4 before correctly finding knowledge quite often. The Spanish Inquisition is a shining example. Or communism - in its original inception, it was supposed to be a utopian paradise where everyone does what work is necessary, and enjoys fair benefits therefrom.

I suspect that a common failure mode is that one fails to take into account that many people are doing that which they are doing because they are quite happy to do it. They've smoothed out any sharp corners in their lifestyle that they could manage to smooth out, and see little benefit in changing to a new lifestyle, with new and unexpected sharp corners that will need smoothing.

I would therefore recommend being very, very cautious about assuming that one has successfully found sufficient knowledge.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-19T13:14:18.108Z · LW(p) · GW(p)

I agree there's a common failure mode here - I'd be inclined to say it's simple overconfidence, and maybe overestimating your rationality relative to everyone else.

Still, if they're successful...

Replies from: CCC
comment by CCC · 2013-04-19T21:21:15.531Z · LW(p) · GW(p)

Even then, I'd most likely object to their attempts to try to dictate the actions of others; because of the common failure mode, my heuristic is to assign a very strong prior to the hypothesis that they are unsuccessful. Also, trying force has some fairly substantial negative effects; any positive effects of their proposed behaviour change would have to be significant to overcome that.

However, if they are willing to try to change the actions of others through simple persuasion without resorting to force, then I would not object. And if their proposed course of action is significantly better, then I would expect persuasion to work in at least some cases; and then these cases can be used as evidence for the proposed course of action working.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-19T22:52:22.340Z · LW(p) · GW(p)

To be fair, we may have different interventions in mind here. I would also expect someone who genuinely found knowledge to use "soft force", but maybe that's just wishful thinking.

However, if forcing people to do things really helps, I'm all for intervention. Addicts, for example.

Replies from: CCC
comment by CCC · 2013-04-20T12:31:29.097Z · LW(p) · GW(p)

I was thinking armies, secret police, so on and so forth, forcing an entire country to one's will.

However, if forcing people to do things really helps, I'm all for intervention. Addicts, for example.

Hmmm. I hadn't thought of addicts. You make a good point.

I think I might need to re-evaluate my heuristics on this point.

comment by RichardKennaway · 2013-04-17T13:20:10.757Z · LW(p) · GW(p)

It's worth noting that if the person successfully "found knowledge", they are in fact correct (unless it was irrelevant knowledge, I guess.)

This can never be put into practice. A person can try to find knowledge, but there is nothing they can do to determine whether they have successfully found knowledge -- any such attempts collapse into part of trying to find knowledge. There is no way of getting to a meta-level from which you can judge whether your efforts bore fruit. The ladder has no rungs.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-17T13:28:01.595Z · LW(p) · GW(p)

raises eyebrows

You're saying it's impossible for any evidence to change your estimate of whether something will help people?

Replies from: RichardKennaway
comment by RichardKennaway · 2013-04-17T13:42:36.754Z · LW(p) · GW(p)

No, just that while you can try harder to find knowledge, there isn't a separate metalevel at which seeing if you really have knowledge is a different activity.

Replies from: MugaSofer
comment by MugaSofer · 2013-04-19T14:00:22.297Z · LW(p) · GW(p)

If you can receive information that provides strong Bayesian evidence that you're belief is true, how is there "nothing they can do to determine whether they have successfully found knowledge"?

comment by Bugmaster · 2013-04-15T21:12:21.890Z · LW(p) · GW(p)

I don't know that much about Mormonism, but isn't it possible that there are multiple different sects of it, just like there are multiple sects of conventional Christianity, Judaism, Wicca, etc. ? In this case, each member of a sect would see himself as the One True Mormon (tm), and would be technically correct, despite believing in different things than members of other sects.

Replies from: JohnH
comment by JohnH · 2013-04-15T21:44:12.578Z · LW(p) · GW(p)

Mormonism is much more structured then that. There are different sects but those sects are different churches, both of us come from the LDS church, which is the largest and the one that everyone thinks of when they say Mormon (unless they are thinking of the polygamous FLDS).

There are those that call themselves New Order Mormons which are within the LDS church, by which they mean they don't believe in any of the truth claims of the church but like the culture (or something like that, I am sure I am taking what they say out of its "rich contextual setting").

Replies from: Bugmaster
comment by Bugmaster · 2013-04-15T22:50:28.693Z · LW(p) · GW(p)

Thanks, that was informative ! So, I assume that the LDS is managed by the Prophet, similarly to how the Catholic Church is managed by the Pope ? I don't mean to imply that the beliefs and the divine status (or lack thereof) of the two are equivalent, I'm merely comparing their places on the org chart.

Although, now that I think about it, even the Catholics have their sub-sects. For example, while the Pope is officially against contraception, many (if not most) American Catholics choose to ignore that part of the doctrine, and IIRC there are even some nuns actively campaigning to make it more accessible.

Replies from: Nornagest
comment by Nornagest · 2013-04-17T21:17:03.184Z · LW(p) · GW(p)

So, I assume that the LDS is managed by the Prophet, similarly to how the Catholic Church is managed by the Pope ?

If memory serves, the President of the (LDS) Church, his advisors, and the members of the church's senior leadership council (called the Quorum of the Twelve Apostles) all hold the title of prophet -- specifically "prophet, seer, and revelator". That doesn't necessarily carry all the implications that "prophet" might outside of an Mormon context, though. One of the quirks of Mormonism is a certain degree of rank inflation compared to most Abrahamic religions; almost all male Mormons enter what the religion calls an order of priesthood) at the age of twelve, for example, and a second when they reach eighteen.

But yes, for most purposes the President of the Church is loosely equivalent to the Catholic Pope. Things get a little funky as you get into lower ranks: the LDS org chart is much more complicated than the Catholic, with several layers of leadership councils and more titles than I can easily keep straight. Though it hasn't developed the numerous unofficial and semi-official leadership roles that Catholicism has, being a smaller and younger religion.

comment by JohnH · 2013-04-15T21:17:35.045Z · LW(p) · GW(p)

I am mostly just answering direct questions, I am horrible at walking away when questions are asked. Since this conversation is far outside of the norms of the group, I will do so in a private message if atomliner wants to continue the conversation. If he would rather it be public I would be willing to set up a blog for the purpose of continuing this conversation.

comment by MugaSofer · 2013-04-17T12:04:14.303Z · LW(p) · GW(p)

The reason we are having this discussion is because I feel you've characterized me unfairly as "the Ex-Mormon who never really knew his own religion and had no reason to believe in the fringe theories he did". My goal is to support my case that I really was a mainstream Latter-day Saint before I lost my faith. So, you can use your apologetic arguments all you want for whatever idea you have about Mormonism, but if they aren't based clearly in the scriptures (which I studied a great deal), and if they were never taught widely in the Church, then why exactly did I err in not coming to the same understanding as you? I do not think you have any good evidence for why I was an atypical Mormon who was unjustified in believing in the things I did.

This here is an excellent point. I'm pretty sure all religions have "unofficial" doctrines; certainly it would fit my experience. Such doctrines have no bearing on the truth of the "official" doctrines, technically, but they are identified with the religion by believers and unbelievers alike.

That said, while I'm hardly an authority on Mormonism, I would guess your beliefs were more, well, strange than average - simply because your deconversion selects for unconvincing and dissonant beliefs.

comment by JohnH · 2013-04-15T20:06:29.828Z · LW(p) · GW(p)

So, you can use your apologetic arguments all you want for whatever idea you have about Mormonism, but if they aren't based clearly in the scriptures

Please show what I said (excluding the reference to Confucius) is not clearly based in scripture, Numbers 11:29 may be helpful.

would not teach false doctrine?

Yes. that is unreasonable to assume

Do you think that a person who speaks face-to-face with Jesus Christ would then teach his own false ideas to members of Christ's One True Church?

Absolutely, if Jesus says something to a prophet then what Jesus said was correct. What the prophet thinks and communicates in addition to that particular thing has no guarantee of being correct and is very likely to be at least partially incorrect. The prophet will place the words of Jesus in the framework of other beliefs and cultural constructs in the world in which they live. Prophets just as much as anyone else do not receive the fullness at once, meaning that of necessity some of their beliefs (and therefore some of their teachings) will not be correct, excluding Jesus. Prophets are not perfect any more then anyone else is perfect and we are supposed to use the light of the Spirit to discern the truth ourselves rather then follow the prophet without thought or seeking to know for ourselves. In other words, telling people to seek God as to every question is calling them to be prophets.

Why is what he said logical to you?

Because I have not stood in the Divine Council and so I know that not only do I not know the secrets of God I also do not have a complete understanding of faith, repentance, baptism, and the Gift of the Holy Ghost, of loving God or of loving my neighbor as myself, nor will I until, either in this life or the next, I hear the Father say Ye shall have eternal life and receive an end to my faith.

And you are how old?

Why is that relevant? Older than you.

Replies from: atomliner
comment by atomliner · 2013-04-15T20:47:10.738Z · LW(p) · GW(p)

Please show what I said (excluding the reference to Confucius) is not clearly based in scripture, Numbers 11:29 may be helpful.

I apologize. I had thought that you were using the three scriptures I quoted earlier to support the point that the scriptures confirms that atheists can be as happy, healthy, and moral as theists. In actuality, you were using them to describe how blessings come from following the commandments and not just from belief in the first two cases and in the third case you were supporting the idea that God understands it is difficult for people to distinguish truth from error.

The point I made about our conversation still stands, however. Your goal seems to be "Make atomliner look like he didn't believe in things Mormons should" while my goal is "show I was a normal Latter-day Saint before losing my faith".

What the prophet thinks and communicates in addition to that particular thing has no guarantee of being correct and is very likely to be at least partially incorrect. The prophet will place the words of Jesus in the framework of other beliefs and cultural constructs in the world in which they live. Prophets just as much as anyone else do not receive the fullness at once, meaning that of necessity some of their beliefs (and therefore some of their teachings) will not be correct, excluding Jesus.

I have two problems with this. The first is that I do not see any scriptures supporting this view clearly. How was I supposed to know this? No one teaches in church that prophets can teach false doctrine. In my experience with hundreds of active Latter-day Saints, THIS belief is atypical. In fact I just got called out by a bunch of mission buddies for saying this on Facebook, that the prophets can sometimes lead us astray (we were talking about gay marriage), and I got called an apostate outright.

My second problem is that I said false doctrine, not small inaccuracies attributed to translation error. You think that a prophet could speak to Jesus Christ face-to-face and then write up entire discourses on stuff like Adam-God theory, blood atonement, doctrinal racism and affirm boldly that this is the truth to the Saints? God must have a very strange way of picking his prophets, it seems like he would want to call people who wouldn't invent their own ideas and who would simply repeat to the Saints what was said to them by Christ. I mean, does God want the truth expressed accurately or not? Were the prophets really the best people available for this task?? They have a terrib