(Moral) Truth in Fiction?

post by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2009-02-09T17:26:12.000Z · LW · GW · Legacy · 82 comments

A comment by Anonymous on Three Worlds Collide:

After reading this story I feel myself agreeing with Eliezer more on his views and that seems to be a sign of manipulation and not of rationality.

Philosophy expressed in form of fiction seems to have a very strong effect on people - even if the fiction isn't very good (ref. Ayn Rand).

Robin has similar qualms:

Since people are inconsistent but reluctant to admit that fact, their moral beliefs can be influenced by which moral dilemmas they consider in what order, especially when written by a good writer. I expect Eliezer chose his dilemmas in order to move readers toward his preferred moral beliefs, but why should I expect those are better moral beliefs than those of all the other authors of fictional moral dilemmas?

If I'm going to read a literature that might influence my moral beliefs, I'd rather read professional philosophers and other academics making more explicit arguments.

I replied that I had taken considerable pains to set out the explicit arguments before daring to publish the story.  And moreover, I had gone to considerable length to present the Superhappy argument in the best possible light.  (The opposing viewpoint is the counterpart of the villain; you want it to look as reasonable as possible for purposes of dramatic conflict, the same principle whereby Frodo confronts the Dark Lord Sauron rather than a cockroach.)

Robin didn't find this convincing:

I don't think readers should much let down their guard against communication modes where sneaky persuasion is more feasible simply because the author has made some more explicit arguments elsewhere...  Academic philosophy offers exemplary formats and styles for low-sneak ways to argue about values.

I think that this understates the power and utility of fiction.  I once read a book that was called something like "How to Read" (no, not "How to Read a Book") which said that nonfiction was about communicating knowledge, while fiction was about communicating experience.

If I want to communicate something about the experience of being a rationalist, I can best do it by writing a short story with a rationalist character.  Not only would identical abstract statements about proper responses have less impact, they wouldn't even communicate the same thought.

From The Failures of Eld Science:

"...Work expands to fill the time allotted, as the saying goes.  But people can think important thoughts in far less than thirty years, if they expect speed of themselves."  Jeffreyssai suddenly slammed down a hand on the arm of Brennan's chair.  "How long do you have to dodge a thrown knife?"

"Very little time, sensei!"

"Less than a second!  Two opponents are attacking you!  How long do you have to guess who's more dangerous?"

"Less than a second, sensei!"

"The two opponents have split up and are attacking two of your girlfriends!  How long do you have to decide which one you truly love?"

"Less than a second, sensei!"

"A new argument shows your precious theory is flawed!  How long does it take you to change your mind?"

"Less than a second, sensei!"


Sweat was forming on Brennan's back, but he stopped and actually thought about it -


"No sensei!  I'm not finished thinking sensei!  An answer would be premature!  Sensei!"

"Very good!  Continue!  But don't take thirty years!"

This is an experience about how to avoid completing the pattern when the pattern happens to be blatantly wrong, and how to think quickly without thinking too quickly.

Forget the question of whether you can write the equivalent abstract argument that communicates the same thought in less space.  Can you do it at all?  Is there any series of abstract arguments that creates the same learning experience in the reader?  Entering a series of believed propositions into your belief pool is not the same as feeling yourself in someone else's shoes, and reacting to the experience, and forming an experiential skill-memory of how to do it next time.

And it seems to me that to communicate experience is a valid form of moral argument as well.

Uncle Tom's Cabin was not just a historically powerful argument against slavery, it was a valid argument against slavery.  If human beings were constructed without mirror neurons, if we didn't hurt when we see a nonenemy hurting, then we would exist in the reference frame of a different morality, and we would decide what to do by asking a different question, "What should* we do?"  Without that ability to sympathize, we might think that it was perfectly all right* to keep slaves.  (See Inseparably Right and No License To Be Human.)

Putting someone into the shoes of a slave and letting their mirror neurons feel the suffering of a husband separated from a wife, a mother separated from a child, a man whipped for refusing to whip a fellow slave - it's not just persuasive, it's valid.  It fires the mirror neurons that physically implement that part of our moral frame.

I'm sure many have turned against slavery without reading Uncle Tom's Cabin - maybe even due to purely abstract arguments, without ever seeing the carving "Am I Not a Man and a Brother?"  But for some people, or for a not-much-different intelligent species, reading Uncle Tom's Cabin might be the only argument that can turn you against slavery.  Any amount of abstract argument that didn't fire the experiential mirror neurons, would not activate the part of your implicit should-function that disliked slavery.  You would just seem to be making a good profit on something you owned.

Can fiction be abused?  Of course.  Suppose that blacks had no subjective experiences.  Then Uncle Tom's Cabin would have been a lie in a deeper sense than being fictional, and anyone moved by it would have been deceived.

Or to give a more subtle case not involving a direct "lie" of this sort:  On the SL4 mailing list, Stuart Armstrong posted an argument against TORTURE in the infamous Torture vs. Dust Specks debate, consisting of a short story describing the fate of the person to be tortured.  My reply was that the appropriate counterargument would be 3^^^3 stories about someone getting a dust speck in their eye.  I actually did try to send a long message consisting only of


for a thousand lines or so, but the mailing software stopped it.  (Ideally, I should have created a webpage using Javascript and bignums, that, if run on a sufficiently large computer, would print out exactly 3^^^3 copies of a brief story about someone getting a dust speck in their eye.  It probably would have been the world's longest finite webpage.  Alas, I lack time for many of my good ideas.)

Then there's the sort of standard polemic used in e.g. Atlas Shrugged (as well as many less famous pieces of science fiction) in which Your Beliefs are put into the minds of strong empowered noble heroes, and the Opposing Beliefs are put into the mouths of evil and contemptible villains, and then the consequences of Their Way are depicted as uniformly disastrous while Your Way offers butterflies and apple pie.  That's not even subtle, but it works on people predisposed to hear the message.

But to entirely turn your back on fiction is, I think, taking it too far.  Abstract argument can be abused too.  In fact, I would say that abstract argument is if anything easier to abuse because it has more degrees of freedom.  Which is easier, to say "Slavery is good for the slave", or to write a believable story about slavery benefiting the slave?  You can do both, but the second is at least more difficult; your brain is more likely to notice the non-sequiturs when they're played out as a written experience.

Stories may not get us completely into Near mode, but they get us closer into Near mode than abstract argument.  If it's words on paper, you can end up believing that you ought to do just about anything.  If you're in the shoes of a character encountering the experience, your reactions may be harder to twist.

Contrast a verbal argument against the verbal belief that "non-Catholics go to Hell"; versus reading a story about a good and decent person, who happens to be a Protestant, and dies trying to save a child's life, who is condemned to hell and has molten lead poured down her throat; versus the South Park episode where a crowd of newly dead souls is at the entrance to hell, and the Devil says, "Sorry, it was the Mormons" and everyone goes "Awwwww..."

Yes, abstraction done right can keep you going where concrete visualization breaks down - the torture vs. dust specks thing being an archetypal example; you can't actually visualize that many dust specks, but if you try to choose SPECKS you'll end up with circular preferences.  But so far as I can organize my metaethics, the ground level of morality lies in our preferences over particular, concrete situations - and when these can be comprehended as concrete images at all, it's best to visualize them as concretely as possible.  Unless we know specifically where the concrete image is going wrong, and have to apply an abstract correction.  The moral abstraction is built on top of the ground level.

I am also, of course, worried about the idea that stories aren't "respectable" because they don't look sufficiently solemn and dull; or the idea that something isn't "respectable" if can be understood by a mere popular audience.  Yes, there are technical fields that are genuinely impossible to explain to your grandmother in an hour; but ceteris paribus, people who can write at a more popular level without distorting technical reality are performing a huge service to that field.  I've heard that Carl Sagan was held in some disrepute by his peers for the crime of speaking to the general public.  If true, this is merely stupid.

Explaining things is hard.  Explainers need every tool they can get their hands on - as a matter of public interest.

And in moral philosophy - well, I suppose it could be the case that moral philosophers have discovered moral truths that are deductive consequences of most humans' moral frames, but which are so difficult and technical that they simply can't be explained to a popular audience within a one-hour lecture.  But it would be a tad more suspicious than the corresponding case in, say, physics.

I realize that I speak as someone who does a lot of popularizing, but even so - fiction ought to be a respectable form of moral argument.  And a respectable way of communicating experiences, in particular the experience of applying certain types of thinking skills.

I've always been of two minds about publishing longer fiction pieces about the future and its consequences.  Not so much because of the potential for abuse, but because even when not abused, fiction can still bypass critical faculties and end up poured directly into the brains of at least some readers.  Telling people about the logical fallacy of generalization from fictional evidence doesn't make it go away; people may just go on generalizing from the story as though they had actually seen it happen.  And you simply can't have a story that's a rational projection; it's not just a matter of plot, it's a matter of the story needing to be specific, rather than depicting a state of epistemic uncertainty.

But to make shorter philosophical points?  Sure.

And... oh, what the hell.  Just on the off-chance, are there any OB readers who could get a good movie made?  Either outside Hollywood, or able to bypass the usual dumbing-down process that creates a money-losing flop?  The probabilities are infinitesimal, I know, but I thought I'd check.


Comments sorted by oldest first, as this post is from before comment nesting was available (around 2009-02-27).

comment by Norman_Noman · 2009-02-09T17:51:51.000Z · LW(p) · GW(p)

I preferred the superhappy ending, and in fact the story nudged me further in that direction. I guess I don't really get what the big deal about pain and suffering is, there's no physical pain on the internet and it seems to work just fine.

comment by nazgulnarsil3 · 2009-02-09T17:57:37.000Z · LW(p) · GW(p)

three worlds collide would make a decent movie...just have to make the reasoning of the characters more explicit for people unfamiliar with concepts involved.

comment by Andy_McKenzie · 2009-02-09T18:11:12.000Z · LW(p) · GW(p)

Eliezer, one qualm: You consistently bring up mirror neurons and consider it to be obvious prima facie that they are used for action understanding in humans. Unfortunately, most contemporary neuroscientists in the field agree that there is no consistent evidence of this:



That is not to say that humans don't understand other people's actions or that we do not have adequate theory of minds! But it does mean that there is no reason to suspect that those complicated cognitive events can be reduced to simply a group of "mirror" neurons. Ramachandran often mentions them as well, which irks me slightly as well.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2009-02-09T18:12:43.000Z · LW(p) · GW(p)

3WC would be a terrible movie. "There's too much dialogue and not enough sex and explosions", they would say, and they'd be right. And you shouldn't just tack them on, either; sex and explosions should flow out naturally as an indispensable part of the plot.

Andy, consider "mirror neurons" as shorthand for "empathic architecture" rather than implying that the whole thing gets done by a small group of actual neurons a la the "grandmother neuron".

Replies from: Strange7
comment by Strange7 · 2010-04-27T21:03:28.178Z · LW(p) · GW(p)

"There's too much dialogue and not enough sex and explosions", they would say, and they'd be right. And you shouldn't just tack them on, either; sex and explosions should flow out naturally as an indispensable part of the plot.

Bullshit. There's that part about rape having been legalized, and the canonical ending involves a planet being destroyed by a supernova. In a visual medium, the hypothetical discussion about exterminating baby-eaters would be voiceover for a montage of helpless crystalline civilians being hunted down by human infantry with power armor and personal laser cannons, culminating in an orbital bombardment.

comment by Zubon · 2009-02-09T18:37:11.000Z · LW(p) · GW(p)

Eliezer: if you show more of it from the perspective of the Superhappies, dialogue itself takes care of that problem.

comment by Tom_McCabe2 · 2009-02-09T18:41:35.000Z · LW(p) · GW(p)

"3WC would be a terrible movie. "There's too much dialogue and not enough sex and explosions", they would say, and they'd be right."

Hmmm.. Maybe we should put together a play version of 3WC; plays can't have sex and explosions in any real sense, and dialogue is a much larger driver.

comment by Kaj_Sotala · 2009-02-09T18:41:54.000Z · LW(p) · GW(p)

And moreover, I had gone to considerable length to present the Superhappy argument in the best possible light.

Hmm. I felt that while the Superhappy argument was presented in the best possible light, the Superhappy ending wasn't. The non-Superhappy ending was in two parts and contained all kinds of cool things, like the use of emergency flags to manipulate the local prediction markets and more insights into the personalities of the different characters. The Superhappy ending, on the other hand, was just one part and was basically just a pretty dull overview of how everybody considered humanity's future to be horrible.

To me, it felt like a pretty blatant statement of "this is the future that we don't want".

comment by Psy-Kosh · 2009-02-09T19:00:30.000Z · LW(p) · GW(p)

Frankly, what I've kinda on and off wanted to see was someone turn Nick Bostrom's "Fable of the Dragon Tyrant" into a movie. That could, perhaps, actually work. Maybe.

Replies from: MBlume
comment by MBlume · 2012-10-22T20:51:39.308Z · LW(p) · GW(p)

Fable of the Dragon Tyrant would make a good animated short, I think.

comment by Tyrrell_McAllister2 · 2009-02-09T19:12:31.000Z · LW(p) · GW(p)

I think that you make good points about how fiction can be part of a valid moral argument, perhaps even an indispensable part for those who haven't had some morally-relevant experience first-hand.

But I'm having a hard time seeing how your last story helped you in this way. Although I enjoyed the story very much, I don't think that your didactic purposes are well-served by it.

My first concern is that your story will actually serve as a counter-argument for rationality to many readers. Since I'm one of those who disagreed with the characters' choice to destroy Huygens, I'm pre-disposed to worry that your methods could be discredited by that conclusion. A reader who has not already been convinced that your methods are valid could take this as a reductio ad absurdum proof that they are invalid. I don't think that your methods inexorably imply your conclusion, but another reader might take your word for it, and one person's modus ponens is another's modus tollens. Of course, all methods of persuasion carry this risk. But it's especially risky when you are actively trying to make the "right answer" as difficult as possible to ascertain for dramatic purposes.

Another danger of fictional evidence is that it can obscure what exactly is the structure and conclusion of the argument. For example, why were we supposed to conclude that evading the Super-Happies was worth killing 15 billion at Huygens but was not worth destroying Earth and fragmenting the colonies? Or were we necessarily supposed to conclude that? Were you trying to persuade the reader that the Supper-Happies' modifications fell between those two choices? As far as I could tell, there was no argument in the story to support this. Nor did I see anything in your preceding "rigorous" posts to establish that being modified fell in this range It appeared to be a moral assertion for which no argument was given. Or perhaps it was just supposed to be a thought-provoking possibility, to which you didn't mean to commit yourself. You subsequent comments don't lead me to think that, though. This uncertainty about your intended conclusion would be less likely if you were relying on precise arguments.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2009-02-09T19:18:56.000Z · LW(p) · GW(p)

Stories and movies are deeply different media. I'm surprised the transition works as often as it does.

comment by JamesAndrix · 2009-02-09T19:30:54.000Z · LW(p) · GW(p)

I've recently become jaded on the use of fiction due to a nearly opposite line of thought: Fiction is really not important.

There are plenty of real problems and real drama.

Eliezer was not able to write about the experience of being a rationalist merely because he read about the experience of being a rationalist in some other work of fiction. Narrative might be required to convey experience, fiction is not.

It is the truths you are trying to convey, that is what matters. But to this we add invented societies, speculation about future technologies, or future technologies that we know can't exist. These may as well be wizards and sorcery, which is another fine genre for burying important lessons.

The complex and detailed universes inevitably lead to utterly pointless arguments.

Plus, the author will be drawn towards classic myth-forms to make the story better. (And the stories will rise and fall based on their appeal as stories, not the validity or true importance of their lessons.)

I recently watched all of the Star Wars movies with someone who had never seen them. She took Palpatine's description of the Jedi as actually power-seeking to be universe-accurate exposition. How many falsehoods will fiction bury the truth under?

comment by billswift · 2009-02-09T19:39:55.000Z · LW(p) · GW(p)

"Without that ability to sympathize, we might think that it was perfectly all right* to keep slaves."

Nearly all people for thousands of years thought it was perfectly all right to keep slaves. Are you saying they didn't have the ability to sympathize? This is the sort of profoundly ahistorical "thinking" that irritates so many people. Someone who considers his own society's beliefs to be laws of reality when there is obvious historical evidence in the other direction that they never bothered to think about.

Replies from: blashimov, johnlawrenceaspden, MugaSofer
comment by blashimov · 2012-10-21T02:55:30.944Z · LW(p) · GW(p)

My take is that without the ability to sympathize we wouldn't have stopped. Not that we suddenly learned/developed sympathy.

comment by johnlawrenceaspden · 2012-10-23T09:27:08.422Z · LW(p) · GW(p)

I don't think it's perfectly all right to eat meat. And I consider battery farming to be an obscenity which I'd like to see outlawed.

And yet I do eat meat, guiltlessly, and not all of it is free-range ethics-meat.

It occurs to me that I might have trouble explaining this to some future ethicist, and I wonder if that might be the same sort of moral state that the Romans, say, were in.

Replies from: RichardKennaway, MugaSofer
comment by RichardKennaway · 2012-10-23T10:44:10.862Z · LW(p) · GW(p)

It occurs to me that I might have trouble explaining this to some future ethicist, and I wonder if that might be the same sort of moral state that the Romans, say, were in.

It's the same sort of moral state we're all in.

comment by MugaSofer · 2012-10-23T13:33:36.184Z · LW(p) · GW(p)

I mock you from the safety of my vegetarianism, while desperately downplaying who made my trainers.

comment by MugaSofer · 2012-10-23T13:32:34.297Z · LW(p) · GW(p)

In most societies, "slaves" were closer to indentured workers than the modern conception of slavery, which descends from the racist North American slave trade, IIRC.

Replies from: johnlawrenceaspden, DaFranker
comment by johnlawrenceaspden · 2012-10-23T16:38:51.833Z · LW(p) · GW(p)

I think I'd take a Southern cotton plantation over a Roman mine any day. And I never heard that the Confederacy lined their roads with crucified rebels.

Replies from: MugaSofer
comment by MugaSofer · 2012-10-24T12:02:17.028Z · LW(p) · GW(p)

Bear in mind that most slaves didn't go to the mines - and they were often used as a punishment.

I had trouble finding details on the punishment of rebel slaves in the Americas - they appear to have been rarer and more successful in escaping capture compared to Roman slaves - but here's something on "maroons" (self-governing pockets of escaped slaves.)

Maroons and their communities can be seen to hold a special significance for the study of slave societies, for they were both the antithesis of all that slavery stood for, and at the same time a widespread and embarrassingly visible part of these systems. The very nature of plantation slavery engendered violence and resistance, and the wilderness setting of early New World plantations allowed marronage and the ubiquitous existence of organized maroon communities. Throughout Afro-America, such communities stood out as an heroic challenge to white authority, and as living proof of a slave consciousness that refused to be limited by the whites' definition and manipulation of it.

Within the first decade of most colonies' existence, the most brutal punishments had already been inflicted on recaptured rebel slaves, and in many cases these were quickly written into law. An early 18th-century visitor to Suriname reported that,

"...if a slave runs away into the forest in order to avoid work for a few weeks, upon his being captured his Achilles tendon is removed for the first offense, while for a second offense... his right leg is amputated in order to stop his running away; I myself was a witness to slaves being punished this way."

And similar punishments for marronage--from being castrated to being slowly roasted to death--are reported from different regions throughout the Americas.

Marronage on the grand scale, with individual fugitives banding together to create independent communities of their own, struck directly at the foundations of the plantation system. It presented military and economic threats that often strained the colonies to their very limits. In a remarkable number of cases throughout the Americas, whites were forced to appeal to their former slaves for a peace agreement. In their typical form, such treaties--which we know of from Brazil, Colombia, Cuba, Ecuador, Hispaniola, Jamaica, Mexico and Suriname--offered maroon communities their freedom, recognized their territorial integrity, and made some provision for meeting their economic needs. In return, the treaties required maroons to end all hostilities toward the plantations, to return all future runaways, and, often, to aid the whites in hunting them down. Of course, many maroon societies never reached this negotiating stage, having been crushed by massive force of arms; and even when treaties were proposed they were sometimes refused or quickly violated. Nevertheless, new maroon communities seemed to appear almost as quickly as the old ones were exterminated, and they remained, from a colonial perspective, the "chronic plague" and "gangrene" of many plantation societies right up to final Emancipation.

To be viable, maroon communities had to be inaccessible, and villages were typically located in remote, inhospitable areas. In the southern United States, isolated swamps were a favorite setting. In Jamaica, some of the most famous maroon groups lived in "cockpit country," where deep canyons and limestone sinkholes abound but water and good soil are scarce. And in the Guianas, seemingly impenetrable jungles provided maroons a safe haven.

Many maroons throughout the hemisphere developed extraordinary skills in guerrilla warfare. To the bewilderment of their colonial enemies, whose rigid and conventional tactics were learned on the open battlefields of Europe, these highly adaptable and mobile warriors took maximum advantage of local environments. They struck and withdrew with great rapidity, making extensive use of ambushes to catch their adversaries in crossfire. They fought only when and where they chose, relying on trustworthy intelligence networks among non-maroons (both slaves and white settlers), and often communicating military information by drums and horns.


Replies from: johnlawrenceaspden
comment by johnlawrenceaspden · 2012-10-25T11:56:41.104Z · LW(p) · GW(p)

These guys sound pretty heroic, but I don't think they're evidence that the racist transatlantic slave trade was worse than the non-racist Roman world. I'm not an expert on either, though.

Part of what I'm trying to assert is that people are capable of treating other people terribly, even in the absence of theories of racial superiority.

I'm pretty sure that the Romans looked up to the Greeks at the same time as enslaving them. And fairly sure that the Greeks enslaved other Greeks.

But you'd need to know a lot more about the classical world than I do to work out what kinds of racial theories were current.

And maybe they did have foreign groups that they mistreated particularly badly. If we think that xenophobia is a built-in feature of the brain then it would be damned weird if the Romans weren't superiority-complex racists. After all, consider the amount of evidence they had that their system was superior and that the gods loved them.

I'd be surprised if it wasn't worse to be the slave of someone who despises you and your type than the slave of someone who accepts you as a brother.

I just don't think any of this is particularly modern.

And on ethical matters I tend to think that progress is upwards (or at least correlated with per-capita GDP). If we think that the recent past was particularly awful it's usually because we've got better records of it.

So here's a prediction for you: There were things going on in the Dark Ages that were worse than either Roman or early Victorian slavery.

The problem is, I can't think of anything worse. There's something particularly terrible about mass industrial slavery. Maybe some passing atrocitologist can help.

Replies from: DaFranker, Nornagest, fezziwig, MugaSofer
comment by DaFranker · 2012-10-25T15:36:54.238Z · LW(p) · GW(p)

The problem is, I can't think of anything worse. There's something particularly terrible about mass industrial slavery. Maybe some passing atrocitologist can help.

I'm not quite an atrocitologist so I have no idea whether some of these things I can think of were actually ever put into practice, but I can think of lots of things worse. I can also guarantee you with 90% confidence that there's a lot of manga (especially doujinshi) out there that do picture things you'd consider much worse, especially when you delve into the darker circles. Some japanese artists have literally become world-renowned 'experts' on the topic of fictional mass atrocity.

I'm not comfortable discussing specific examples without a wall of spoiler prevention features requiring the viewer to pass a mental fortitude test to view the content. I might have mentioned this before, but I've once had an acquaintance bend down and vomit on the spot upon recounting one of my more horrible nightmares. I try to avoid dishing out such mental damage on unprepared individuals nowadays.

Replies from: johnlawrenceaspden, MugaSofer
comment by johnlawrenceaspden · 2012-10-25T17:23:11.016Z · LW(p) · GW(p)

Touché. I meant something that was likely to have actually happened on a fairly large scale.

comment by MugaSofer · 2012-10-26T08:34:06.033Z · LW(p) · GW(p)

Now I'm all curious.

To hear things so bad they make unprepared listeners spontaneously vomit, not to hear things worse than slavery. There are plenty of those, they just tended not to catch on.

comment by Nornagest · 2012-10-25T18:26:49.910Z · LW(p) · GW(p)

So here's a prediction for you: There were things going on in the Dark Ages that were worse than either Roman or early Victorian slavery. The problem is, I can't think of anything worse. There's something particularly terrible about mass industrial slavery. Maybe some passing atrocitologist can help.

Well, I'm not exactly an atrocitologist, but I have studied the early medieval period in some detail. There are some problems in comparing it to other periods, especially in subjective terms -- the Dark Ages were called "dark" precisely because they left a relative dearth of subjective material -- but here's what I can remember off the top of my head.

There was a widespread slave trade, beginning during or before Roman times and ending in Britain around 1100 AD. It was not racially motivated or justified, as we'd understand race; slaves came from all the European ethnic groups, including those of their holders. Taking slaves seems to have been more common in conflicts between ethnic groups, however. Unransomed captures in wartime and freemen who fell into various kinds of legal trouble could both become slaves; the former seem to have been more common. They generally could be bought and sold and didn't have legal independence. The law codes of the time prescribed punishments for mistreating other people's slaves but not your own.

Slave labor was not usually highly concentrated or regimented (there were, for example, no galley slaves in that period); slaveholders came from all free social classes, and slaves performed much the same work as freemen (though usually the harder and dirtier shares of it, where division of labor was possible). At the time of the Domesday Book, slaves made up about 9% of the population.

From what I know of it, this seems more comparable to Roman slavery than to the trans-Atlantic slave trade. Early medieval Europe was a poorer place than either Rome or the early modern colonies, and its people probably led harsher lives, but in social terms I don't see much in the way of unique awfulness.

comment by fezziwig · 2012-10-25T19:50:56.714Z · LW(p) · GW(p)

I wish I'd thought to pick 'Atrocitologist' as a screen name. Oh well.

I can't think of any medieval atrocities comparable in scope to those of either the Roman or Victorian eras. But I don't think that has anything to do with philosophy or tolerance, it's just that Rome and pre-Victorian England were a lot more powerful and effective than any of the intermediate governments, and so were able to achieve greater scope than e.g. Poland ever could.

But to your more general point: modern racism is just a special case of the human tendency to define ingroup/outgroup divisions, right? It's ok to enslave Them, because they're not Us. That finding is extremely robust through history: Greeks enslaved other Greeks (but they called themselves Spartans and Helots), Italians enslaved other Italians (but the victims were never Roman citizens so it didn't count), the Jews wiped out the Amelikites (they worshipped the wrong gods, what can you do?) and French nobles ruled over French serfs (but you can't compare a noble to a serf).

Replies from: MugaSofer
comment by MugaSofer · 2012-10-26T08:19:33.844Z · LW(p) · GW(p)

Italians enslaved other Italians (but the victims were never Roman citizens so it didn't count)

Romans could be sold into slavery to pay off their debts.

The Romans were reletively free of out-group hostility - they felt the barbarians outside the empire were savages, but they tended to absorb local power structures and religions, granting the local nobles (if they cooperated) Roman citizenship, (which was more exclusive than, say, American citizenship,) and while there was some generic snobbery there does not appear to be any belief that non-Romans were inherently inferior. Once they joined the empire, they gained all the rights and privileges of your average Roman (including protection from those barbarian savages over the hill.)

comment by MugaSofer · 2012-10-26T08:36:33.922Z · LW(p) · GW(p)

I don't think they're evidence that the racist transatlantic slave trade was worse than the non-racist Roman world.

They aren't. However,

"...if a slave runs away into the forest in order to avoid work for a few weeks, upon his being captured his Achilles tendon is removed for the first offense, while for a second offense... his right leg is amputated in order to stop his running away; I myself was a witness to slaves being punished this way."

And similar punishments for marronage--from being castrated to being slowly roasted to death--are reported from different regions throughout the Americas.

grants context to your statement that "I never heard that the Confederacy lined their roads with crucified rebels."

we think that xenophobia is a built-in feature of the brain

Xenophobia and racism are different things.


Is that ... a Culture ship name?

comment by DaFranker · 2012-10-23T17:13:36.285Z · LW(p) · GW(p)

I'm not sure that holds. In my model and priors, the racist NA slave-owners are much less likely to do certain categories of negative things, and are actually on the higher end of the nice-slavery scale on account of them usually being too disgusted or other-negative-feelings to bother doing anything to them but the occasional light no-contact beating and lots of forced labor.

For instance, considering that I'd guesstimate roughly 10% of Roman Empire slaves to be privately-owned females, and that of those a certainly non-negligible amount would be attractive to their owners or people-the-owner-wants-to-be-friends-with somehow, then it's not that far-fetched to guesstimate that around 5% of Roman Empire slaves were, in fact, used often or mostly for sexual purposes. This obviously includes rape and torture and whatever other fun things the owners/friends-of wanted to do.

I'd also presume that it was much easier to start off a roman sex-and-food orgy with slaves than with free females that are otherwise just interested in having such an orgy considering what I know of cultural gender expectations/roles for that culture and of the numbers of such free vs slave women.

I've also heard that the ancient Chinese had some pretty sick ideas (both horrible and genius at the same time) of what to do with slaves during their constant warfare, including but not limited to: Training items (e.g. "These slaves have wooden swords, but we've broken half their fingers and wounded their arms. Today, you will be training how to effectively cripple and immobilize an enemy with your ranged weapons before they get within sword range!"), battlefield entertainment, emergency food reserves, literal meat walls (no, not putting them on the front lines to fight; literally slaying and piling them at a particular spot just to hinder the enemy movements and cause psychological damage), and various forms of experimentation, testing of poisons/herbs, or whatever-the-owner-felt-like-doing-to-them.

Likewise, slave life in ancient Babylon-and-nearby or around the middle-east during the european middle age and renaissance don't sound all that attractive, nor nearly what I'd qualify as "indentured worker".

The most striking indentured-worker-like slaves I can think of come from some limited data hinting that Egyptian slaves during the ages of grand pharaohs were mostly pretty well treated and considered important belongings equivalent to how we'd value today a car with integrated PC if that was something common.

Of course, a lot of the above is from very limited evidence and there's surely a lot of just-so mixed in, but considering the current state of modern slave trade and black market (forced) sex trade around the world and its prevalence (and the amounts of money involved), I would strongly favor hypotheses that contain similar horrible conditions in most slave cultures in history on account of not knowing of any particular factor or change in human nature and societies that would suddenly make it more common or likely in our current world and cultures.

Replies from: TimS, MugaSofer
comment by TimS · 2012-10-23T17:37:54.443Z · LW(p) · GW(p)

This is true, but chattel slavery (the type of slavery practiced in the American South) is not the only kind of slavery practiced in history. Some slave cultures (Greek and Roman in particular) allowed a real possibility of manumission into freedom, or even citizenship.

Assuming chattel (or ethnicity-based) slavery was the only kind is a failure mode in the general public.

Edit: I'm not sure where you got this idea:

the racist NA slave-owners are much less likely to do certain categories of negative things, and are actually on the higher end of the nice-slavery scale on account of them usually being too disgusted or other-negative-feelings to bother doing anything to them but the occasional light no-contact beating and lots of forced labor.

The brutality of US chattel slavery varied from place to place and time to time, but there's mixed evidence at best for genuine sympathy for slaves among the slave-owners or free folks. A lot of pro-slave court cases are successful attempts by slave-owners to punish slave-renters who mistreated the property or free bystanders engaging in destructive "vandalism".

Replies from: DaFranker
comment by DaFranker · 2012-10-23T18:19:54.103Z · LW(p) · GW(p)

Re. that quote, I was referring mostly to the kinds of abuse I cite in example in the rest of the comment.

From what I know, it wasn't particularly common for American South slave-owners to specifically select for attractive female slaves and routinely rape or otherwise abuse them for personal pleasure - having sex with "niggers" was, as I understand it, a deep wrongdoing.

I also don't imagine the american slave-owners of that culture engaging in all that many orgies or the various practices of viking naval slaves or ancient chinese generals.

Replies from: TimS, MugaSofer
comment by TimS · 2012-10-23T19:17:55.928Z · LW(p) · GW(p)

I'm not sure where you are getting the violent sex aspect of slavery - I'm not an expert, but I'm not aware of that as a widespread practice. Treating slaves' lives as cheap (like your examples from Chinese history) doesn't necessarily imply violent orgies. In my mental model, sex slaves are mentally coerced, not physically coerced. Assaultive rape isn't the first image that appears in my head for sex-with-slaves.

And US chattel slavery is weird about sex with slaves. For example, Sally Hennings is not an usual story, even if that kind of conduct was considered improper. Further, the idea of slaves as sexually charged beings continues to run through current American ideas about US blacks. Consider the trope that black men have above average sized penii.

Replies from: DaFranker
comment by DaFranker · 2012-10-23T19:49:49.592Z · LW(p) · GW(p)

In my mental model, sex slaves are mentally coerced, not physically coerced. Assaultive rape isn't the first image that appears in my head for sex-with-slaves.

This is what I have in mind when I say "routinely rape or otherwise abuse them for personal pleasure". I'm not sure either where I implied a violent or assaultive aspect. Playing on the helplessness of the victim and various forms of mindbreak or psychological coercion is clearly the dominant tactic in most cultures of sexual slavery as far as I'm aware.

Replies from: TimS
comment by TimS · 2012-10-25T14:47:42.101Z · LW(p) · GW(p)

I'm not sure either where I implied a violent or assaultive aspect.

Well, from the great grandparent:

This obviously includes rape and torture and whatever other fun things the owners/friends-of wanted to do.

In general, strangers-to-the-bond were not permitted the same liberties with slaves as owners.

Replies from: DaFranker
comment by DaFranker · 2012-10-25T15:23:40.838Z · LW(p) · GW(p)

Ah. I'm doubtful that the line between owners doing what they wanted with their slaves and their friends doing the same to same slaves was that clear-cut or actually upheld, but it makes sense now that I think about it - it certainly makes sense that strangers would avoid damaging the slaves of others or when in doubt, since that would legally amount to high vandalism and property damage.

My conception of "torture" is very large. Sure, that includes strapping someone to a chair and flogging them, but I'm not so skewed as to believe such things were commonly practiced onto most categories of slaves unless there were special circumstances.

Things like promising slaves a real meal if they work twice as hard for the day and then giving them authentic dog feed once they're done counts as torture within this enlarged label. Such things were, I'm given to understand, very widespread among various slavery cultures, sometimes as a form of entertainment.

To my understanding, they're currently very widespread in current 2012 slavery (though apparently most people nowadays use the term "human trafficking", which obscures from discussion what actually happens once the person has been trafficked).

Replies from: TimS
comment by TimS · 2012-10-25T15:56:56.607Z · LW(p) · GW(p)

There's a relatively recent post about overly expansive definitions and why the changes in connotation they create are bad - but I can't find.

I will say that a wildly nonstandard definition of a term increases inferential distance. And there are labels for the concepts you want to gesture towards - expanding the reach of the label "torture" is not necessary.

Replies from: DaFranker
comment by DaFranker · 2012-10-25T16:58:56.120Z · LW(p) · GW(p)

I wasn't aware of better labels that could be used in nontechnical discussion, and I can't think of any at the moment.

In practice I've used this expanded "torture" label in discussions with various sorts of people both specialized and not (though this does not include professional sociologists, historians or moral philosophy experts), and found that it was usually understood from context without additional input, and otherwise a single extensional example (like the typical bullying case where a couple of kids take someone's bag and throw it around while the owner helplessly runs around trying to get their stuff back) was sufficient for them to make the remaining inferences.

Granted, this may in part be due to body language and other unspoken information channels, and I have transmitted neither this nor any examples before the grandparent. I also didn't think much of it at the time of writing, so I'll agree that it was a mistake, since I could just as well have rationalized in this manner even if I did not observe it as a successful label in previous instances.

Replies from: TimS
comment by TimS · 2012-10-25T17:26:11.423Z · LW(p) · GW(p)

That makes sense.

I'm just trying to distinguish between behavior a "law-abiding" slave might never encounter in her lifetime, as opposed to what she probably deals with on a weekly basis (on average - and that's not intended to count the standard "get back to work" instruction). A substantial part of the confusion in this conversation was my reading your words as asserting the former behavior (archetypal torture) was more frequent than historically occurred. My understanding was that behavior towards escape attempts and such (most of which would qualify as torture) was very different than "ordinary" treatment of slaves in the American South.

A phrase like emotional abuse or just abuse might be reasonable label, depending on what other connotations it brings to the conversation.

Replies from: DaFranker
comment by DaFranker · 2012-10-25T17:36:25.063Z · LW(p) · GW(p)

A phrase like emotional abuse or just abuse might be reasonable label, depending on what other connotations it brings to the conversation.

Indeed, in retrospect that should have been an obvious choice. I've been conditioned by the French word "abus" (also the verb "abuser") which carries different connotations and which in common usage would generate a lot more misunderstanding, so I tend to underuse the words "abuse" and its subsets.

comment by MugaSofer · 2012-10-24T11:52:05.141Z · LW(p) · GW(p)


Raping slaves was common, and was often prompted by racism - after all, if the kid's half-white he'll make a better slave, wont he? No-one thought they were unattractive, just "inferior".

comment by MugaSofer · 2012-10-24T11:49:37.400Z · LW(p) · GW(p)

In my model and priors, the racist NA slave-owners are much less likely to do certain categories of negative things, and are actually on the higher end of the nice-slavery scale on account of them usually being too disgusted or other-negative-feelings to bother doing anything to them but the occasional light no-contact beating and lots of forced labor.

It's hard to get estimates for beatings - there appears to have been a great deal of variation - but let's just say that racism is unlikely to decrease beatings. And American slaves were definitely raped. A lot. Sometimes because they were considered inferior - after all, if the kid is half white, they'll make a better slave, right?

Roman slaves are the ones I know most about. And while they had essentially no rights, there was no concept of them being inherently inferior - they simply had lost their freedom, whether selling it to cover debts or having it taken as spoils of war. They could not be identified visually, and if freed were the equal of any Roman. Roman slaves gradually acquired more rights as time went on.

The master's power over the slave was called (dominica potestas), and it was absolute. Torture, degradation, unwarranted punishment, and even killing a slave when he was old or sick, in the eyes of the law, slaves were property who could not legally hold property, make contracts, or marry, and could testify in court only under torture. The death of his master did not free a slave. Under the Empire laws were passed stating that a slave could not be sold to fight wild beasts in the amphitheater; he could not be put to death by his master simply because he was old or ill; if her were 'exposed', or turned out on the streets to die, his was freed by the act; and he could not be killed without due process of law. But these laws were generally disregarded, and only the influence of Christianity changed the condition of slaves for the better.

Romans were not a kindly people, but they did not often forget that a slave was valuable property. Much depended on the individual master. Vedius Pollio, notorious for cruelty, once ordered a slave to be thrown alive into a pond as food for the fish because he had broken a goblet. But Cicero had great affection of his slave Tiro. The Elder Cato tells us something about the treatment of farm slaves. He held that slaves should always be at work except for the hours - few enough at best - allowed them for sleep. Slaves were not well fed, but it must be remembered that the diet suggested by Cato (grain, fallen olives or salf fish and sour wine) was very similar to that of the poor Romans. A slave received a tunic every year and a cloak and pair of wooden shoes every two years. Worn-out clothing was returned to the slave manager to be made into patchwork quilts.

If a slave escaped, he had to live the life of an outlaw, with organized bands of slave hunters on his track. A fugitive slave was a criminal, for he had stolen himself. If he was caught, he was branded on the forehead with the letter F, for fugitivus, and sometimes he had a metal collar riveted around his neck. One of these collars, preserved at Rome, says in Latin, "I have run away. Catch me. If you take me back to my master Zoninus, you'll be rewarded".

A slave could not legally own property, but he often had peculium, unofficial possessions. Often an industrious, thrifty slave could scrape together a little fund of his own if his master permitted it. City slaves had more chance to do this, collecting tips from his master's friends and guests or receiving presents from students if he was a teacher. Sometimes a master would allow a slave to have a trade and keep part of the earnings. A thrifty slave's ultimate goal was to buy his freedom. Sometimes a slave would buy his own slave to hire out. A slave of a slave was called a vicarius. A slave's property went to his master upon the slave's death.

Slaves were often punished. The most common one for neglect of duty or petty misconduct was a beating or a flogging with a lash (called a flagrum or a flagellum). Sometimes slaves were punished by having to wear a heavy forked log around his shoulders with his neck in the fork and his arms fastened to the ends projecting in front. This is where the term of abuse furcifer came from. Minor punishments were inflicted at the order of the master or his manager by a fellow slave, called for the time carnifex(executioner). Occasionally a slave would be assigned to harder labour than he was accustomed to. Utterly incorrigible slaves were sold to be gladiators. Punishments were severe for actual crimes, always a possibility since slaves were so numerous and had such free access to their master. Nothing was so much dreaded throughout all Italy as an uprising of slaves. For an attempt on a master's life or for taking part in an insurrection, the penalty was death for the criminal and his family in a most agonizing form - crucifixion. Pompey erected six thousand crosses along the road to Rome, each bearing a survivor of the final battle in which their leader, Spartacus, fell. The word crux (cross) was used amoung slaves as a curse, especially in the expression [I] ad [malam] crucem ([Go] to the [bad] cross).

A slave might buy his freedom, or he might be freed as a reward for faithful service or some special act of devotion. A formal act of manumission often took place before or praetor, but it was only necessary for his master to declare him free before witnesses. A new-made freedman set on his head the cap of liberty. A freedman was called libertus as an individual or in reference to his master, and libertinus as one of a class. His former master became his patron. [source]

Some links:

http://www.thetalkingdrum.com/wil.html http://www.spartacus.schoolnet.co.uk/USASpunishments.htm http://en.wikipedia.org/wiki/Treatment_of_slaves_in_the_United_States#Punishment_and_abuse http://en.wikipedia.org/wiki/Fugitive_slave

Roman slaves: http://en.wikipedia.org/wiki/Slavery_in_ancient_Rome http://www.roman-colosseum.info/roman-life/slave-punishment.htm http://www.mlahanas.de/Greeks/LX/SlavesRomanEmpire.html

Replies from: DaFranker
comment by DaFranker · 2012-10-25T14:26:19.015Z · LW(p) · GW(p)

Much thanks for all the extra data! That was some interesting reading.

comment by billswift · 2009-02-09T19:43:10.000Z · LW(p) · GW(p)

"The complex and detailed universes inevitably lead to utterly pointless arguments."

This is actually an argument for using fiction. Real situations are more complex and much, much more likely to result in peripheral arguments than fictional situations.

comment by frelkins · 2009-02-09T20:07:34.000Z · LW(p) · GW(p)


"Nearly all people for thousands of years thought it was perfectly all right to keep slaves"

And many still do today - for example, Shari'a endorses slavery. Our Western values are far from universal and cannot be taken for granted.

comment by Z._M._Davis · 2009-02-09T20:42:22.000Z · LW(p) · GW(p)

Tom: "Hmmm.. Maybe we should put together a play version of 3WC [...]"

That reminds me! Did anyone ever get a copy of the script to Yudkowski Returns? We could put on a benefit performance for SIAI!


comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2009-02-09T20:52:30.000Z · LW(p) · GW(p)

And perhaps the current slavers would change their minds if they read the right book, and perhaps not - more probably not, I think, without other changes as well. As I noted in the post text, the mirror neurons do have an off switch. It might take some abstract argument to turn them back on. Or it might take a "slave" rescuing their daughter in real life, instead of fiction. Maybe even that wouldn't do it. Maybe their and my reflective equilibria are so far apart that they can't be called by the same word "right".

Nonetheless - Uncle Tom's Cabin had an impact. Historically speaking.

It's easy to talk about how the Other is an alien monster who will just refuse to be persuaded by anything. I can't persuade the invincibly obstinate and despicable image of them that exists in your heads. Reality might be another story.

comment by HughRistik · 2009-02-09T21:10:59.000Z · LW(p) · GW(p)

Tyrrell said:

Nor did I see anything in your preceding "rigorous" posts to establish that being modified fell in this range It appeared to be a moral assertion for which no argument was given.

Yeah. Eliezer, in your story, being modified just didn't seem bad enough to be obviously preferable to killing 15 billion people. This creates moral ambiguity that is great in a story, but not if you wanted to communicate a clear moral.

The way the story was presented, I was think "humanity without suffering, and having to eat non-sentient babies?... Is that really so bad to justify killing 15 billion people?" Now, as a reader of Overcoming Bias, I know that Value is Fragile, and that scaling up the human brain is a highly risky proposition that that the brain is not designed for. So, the end result of the Superhappy proposal would not be "Humans minus suffering, eating non-sentient babies." It would not be human at all.

Superhappies can't just surgically remove negative emotions and pain from our brains and leave everything else untouched. It would be likely the Superhappies would make a first pass and remove suffering, but the neurochemical changes would drive us all insane (happily insane). The Superhappies would then have to make another pass to stabilize our brains, which would involve messing with who-knows-what. But stabilize us towards what? The Superhappies can't know the "right way" to make a sane human brain which doesn't experience suffering, because no such thing exists. If the Superhappies were ever a loss of what to do, then would probably just alter us in the direction of their own values and psychology. The end result of the Superhappy's working on us would probably think like a Superhappy, except with some token human values.

Even if the Superhappies were able to strip away human pain without mishap, there could be negative unintended consequences. If you remove negative emotions, you would actually disinhibit a lot of antisocial human behavior due to the loss of shame and guilt. Then the Superhappies would have to remove any aggressive or antisocial impulses we have, resulting in even more changes, which would all lead to a risk of insanity, or other problems that require even more "fixes."

Any modification the Superhappies make is only going to lead to consequences which result in even more modifications, which have their own consequences. When does this stop? I think the answer is that it doesn't stop until the product is much more Superhappy than it is human. (If instead the Superhappies were to let the humans be in charge of modifying themselves, then a higher degree of continuity with past humanity might be preserved.)

So Eliezer, you and I know the potential pitfalls of modifying humans, but since the story doesn't show them, the Superhappy proposal looks overly attractive, and the humans who resist it look excessively close-minded and trigger-happy in killing 15 billion of their own kind in order to resist something that just doesn't seem as bad (in the context of the story). To truly complete the story to show what you want it to show, you could have a second part of the normal ending that shows exactly why the Superhappy proposal is so bad based on your writings about the riskiness of brain modification.

comment by JamesAndrix · 2009-02-09T21:39:21.000Z · LW(p) · GW(p)

billswift: But those peripheral argument will still be about things that in some sense matter, as opposed to say, midichlorians.

comment by Sideways2 · 2009-02-09T22:36:56.000Z · LW(p) · GW(p)


Speaking as a new reader of Overcoming Bias myself--I think that the sort of people who read this blog are more likely to miss how dangerous the Superhappies are, because we've considered ways that human suffering could be reduced or eliminated while still letting humans develop properly. Then, when people who already have ideas about how to reduce suffering read that the Superhappies want to eliminate suffering, they assume that the Superhappies' plans are the same as their own. (I'm not sure if this is a previously discussed and named bias, but it sure ought to be.)

As far as I can tell, the Superhappies don't care about proper human development, and are not even curious as to what it is. They want us to be happy; being "good people" doesn't enter into it. I'd say the Superhappies are "paperclip maximizers" for happiness-- though their idea of happiness is more complicated than a paperclip, the same principle is at work.

I would have said that the Superhappy proposal to find a happy middle ground between their values and the Babyeaters' by having everyone eat thousands of nonsentient babies was a preposterous straw-man for moral relativists, if that proposal hadn't actually been even more preposterously defended in the comments. Even if it's morally neutral to eat thousands of nonsentient babies, doesn't it seem... well, kind of ridiculous?

Which leads me to a point about the subject of this post, one that I don't think has been brought up yet: sometimes, people understand something more easily and more completely if they can see an example of it. Which is easier, to explain to someone what a cracker is, or to just show them a cracker? It's not practical to build a paperclipper and show it to everyone -- and that's where fiction comes in.

Replies from: johnlawrenceaspden
comment by johnlawrenceaspden · 2012-10-23T09:50:30.369Z · LW(p) · GW(p)

Isn't any rational agent a paperclip maximiser for something? I thought that was what 'rational' was supposed to mean round here.

Replies from: MugaSofer
comment by Yvain2 · 2009-02-09T23:54:34.000Z · LW(p) · GW(p)

Uncle Tom's Cabin is not a valid argument that slavery is wrong. "My mirror neurons make me sympathize with a person whose suffering is caused by Policy X" to "Policy X is immoral and must be stopped" is not a valid pattern of inference.

Consider a book about the life of a young girl who works in a sweatshop. She's plucked out of a carefree childhood, tyrannized and abused by greedy bosses, and eventually dies of work-related injuries incurred because it wasn't cost-effective to prevent them. I'm sure this book exists, though I haven't personally come across it. And I'm sure this book would provide just as emotionally compelling an argument for banning sweatshops as Uncle Tom's Cabin did for banning slavery.

But the sweatshop issue is a whole lot more complex than that, right? And the arguments in favor of sweatshops are more difficult to put into novel form, or less popular among the people who write novels, or simply not mentioned in that particular book, or all three.

The problem with fiction as evidence is that it's like the guy who say "It was negative thirty degrees last night, worst snowstorm in fifty years, so how come them liberals are still talking about 'global warming'?". It cuts off a tiny slice of the universe and invites you to use it to judge the entire system.

But I agree that fiction is not solely a tool of the dark side. Eliezer's comment about it activating the Near mode thinking struck me as the most specifically useful sentence in the entire post, and I would like to see more on that. I would also add one other benefit: fiction drags you into the author's mindset for a while against your will. You cannot read the book about the poor girl in the sweatshops without - at least a little - cheering on the labor unions and hating the greedy bosses, and this is true no matter how good a capitalist you may be in real life. It confuses whatever part of you is usually building a protective shell of biases around your opinion, and gets you comfortable with living on the opposite side of the argument. If the other side of the argument is a more stable attractor, you might even stay there.

...that wasn't a very formal explanation, but it's the best way I can put it right now.

comment by Anonymous48 · 2009-02-10T00:53:24.000Z · LW(p) · GW(p)

Not enough sex and explosions in 3WC? Are you joking?

Oh, and it would be easier to find someone make it into a good visual novel rather than a good movie.

comment by Robin_Hanson2 · 2009-02-10T01:33:31.000Z · LW(p) · GW(p)

Yvian, I warned against granting near-thought virtues to fictional detail here. I doubt Uncle Tom's cabin would have persuaded many slave holders against slavery; I expect well-written well-recommended anti-slavery fiction more served to signal to readers where fashionable opinion was moving.

comment by Vladimir_Nesov · 2009-02-10T01:41:14.000Z · LW(p) · GW(p)

@Yvain: Mathematical proof is a valid argument, even if it doesn't contain any information, what was true remains so.

Fiction isn't supposed to act as evidence, it's supposed to place you in a specific focus of attention, where you resolve your own questions for yourself, from evidence you already hold. It doesn't explicitly state abstractions which you are supposed to learn in order to master new thoughts, reinterpret old data, or bind existing morals. It invites you to invent abstractions on a given topic for yourself.

Of course, all the usual biases will haunt you no less than in real life, plus the bias to interpret fiction as literal evidence, and they can be exploited to derail you just as it happens in the real life, only with more control. Although, control in fiction is still like programming: omnipotence without omniscience, where ability to manipulate the story doesn't always come with a way to efficaciously bias the reader.

comment by Cabalamat2 · 2009-02-10T02:06:21.000Z · LW(p) · GW(p)
Just on the off-chance, are there any OB readers who could get a good movie made?

If machinima counts as "a good movie" you might want to talk to Hugh Hancock (I've no idea if he reads OB, but based on his other interests he may well do).

comment by Robin_Hanson2 · 2009-02-10T02:42:04.000Z · LW(p) · GW(p)

Eliezer, as I indicate in my new post, the issue isn't so much whether you the author judge that some fiction would help inform readers about morals, but whether typical readers can reasonably trust your judgment in such things, relative to the average propaganda content of authors writing apparently similar moral-quandary stories.

comment by TreeFrog · 2009-02-10T02:56:35.000Z · LW(p) · GW(p)

I know that Tucker Max, whose movie I Hope They Serve Beer in Hell will be out sometime this year, has read this site since July of 2007 at least. He's actually how I discovered Overcoming Bias.

He's said numerous times that Eliezer would be absolutely fantastic if his posts weren't so ridiculously long and wandering at times.

He'd be a good person to talk to about making a movie (since his own was designed specifically to avoid that "dumbing-down process") and is probably going to make several tens of millions over the next few years.

comment by TGGP4 · 2009-02-10T03:19:35.000Z · LW(p) · GW(p)

Why should we believe there are "moral truths"? And why are the rules so different with regard to physics? What other topics have a standard more like morality than physics?

I agree with Yvain. The mirror neuron argument was just shoddy. After acknowledging that the science didn't necessarily support your point about them, you then said that doesn't matter. If the truth of an argument is irrelevant, why bring it up at all? Doesn't such an argument falling back on "deeper truth" have the same weaknesses as the religious/mystical in their attempts to avoid falsification?

This is an idea that I think is plausible, although it might be false: Uncle Tom's Cabin was more an epiphenomenon in the demise of slavery than a cause. It is an easy focal point to think of, and so we associate the end of slavery with it. If the book had failed (perhaps through having a lousy publisher or distribution) we would instead point to something else whose fame has been displaced in our own history by Stowe's novel.

comment by billswift · 2009-02-10T04:07:41.000Z · LW(p) · GW(p)

"But those peripheral argument will still be about things that in some sense matter, as opposed to say, midichlorians."

Of course they are. That's why they distract so strongly from the central point.

Why would anyone argue about midi-chlorians? The "explanation" of Jedi powers in that movie detracted from the Star Wars universe.

comment by JamesAndrix · 2009-02-10T04:59:36.000Z · LW(p) · GW(p)

Of course they are. That's why they distract so strongly from the central point.

Even if they distract more from the central point, theya re still real. They still have some potential relevance to the reader.

I think it would be a disservice to train someone up in the arts of rationality only to have most of their thoughts revolve around the facts of some fantasy universe.

Also, just because our universe is more detailed than fictional ones, doesn't mean that all of that detail has to be available to the reader. We can offer simplified descriptions of situations.

comment by michael_vassar3 · 2009-02-10T06:49:34.000Z · LW(p) · GW(p)

Eliezer: It may be worth noting that SIAI just hired a new president FROM a branch of the film industry who has some familiarity with the sort of tax laws that can make indie movies a good investment even when expected value appears negative, and that SIAI's largest donor is the producer of an excellent movie about the marketing of cigarettes.

Other than that.

I agree with Kaj I really like Hugh's point I don't think 3WC or Dragon Tyrant work as movies. I don't know what Eliezer's got however WRT stories.

comment by Michael_Vassar5 · 2009-02-10T08:20:30.000Z · LW(p) · GW(p)

Tree Frog: do you know Tucker and are you suggesting that I speak with him? That's basically my job after all.

comment by Norman_Noman · 2009-02-10T11:25:40.000Z · LW(p) · GW(p)

I still don't see the actual practical benefit of suffering. I've lived a very sheltered life, physically and emotionally. I've never needed stitches, never had my heart broken, I've always been pretty low-key emotionally, and I don't feel like I'm missing anything.

Besides, what are we going to do NEXT time we run into a more advanced race of aliens? I suppose we can just keep blowing up starlines, but what happens if they get the jump on us, like the superhappies got the jump on the babyeaters? It seems like we need powerful allies much more than we need our precious aches and pains.

comment by V.G. · 2009-02-10T11:42:08.000Z · LW(p) · GW(p)


I am very much inclined to analyze your articles because you are indeed very enthusiastic about your theories, which is a rarity these days. On the link there’s a Wordle tag cloud picture of your article.

As you can see, there’s a lot of “argument(s)”,”moral”, “abstract”, “fiction’, and a somewhat humble “experience”.

To the point – fiction in the literary domain is often a method for implying moral concepts. But fiction is, above all, an invitation to imagine. There is a catch. We can imagine a setting, a world, a relationship. But we sometimes cannot imagine that this setting, or world, or relationship, is morally justified, simply because we have an elaborate moral hierarchy to begin with.

Another aspect is that fiction is a theory we know is false, but useful. Beware, Yudkowski – fiction is typically useful when it relates to things that exist or could exist, but only if we are able to observe them one day.

Hence a problem with science fiction – it typically uses big time spans to make reading them books interesting. In the end, as we very well know, science fiction is either too far sighted or too short sighted, but not really useful.

So what the hell am I talking about? Eliezer, in your work there are many useful ideas. “Three Worlds Collide” is more like entertainment.

comment by Yvain2 · 2009-02-11T00:16:37.000Z · LW(p) · GW(p)

@Robin: Thank you. Somehow I missed that post, and it was exactly what I was looking for.

@Vladimir Nesov: I agree with everything you said except for your statement that fiction is a valid argument, and your supporting analogy to mathematical proof.

Maybe the problem is the two different meanings of "valid argument". First, the formal meaning where a valid argument is one in which premises are arranged correctly to prove a conclusion eg mathematical proofs and Aristotelian syllogisms. Well-crafted policy arguments, cost-benefit analyses, and statistical arguments linked to empirical studies probably also unpack into this category.

And then the colloquial meaning in which "valid argument" just means the same as "good point", eg "Senator Brown was implicated in a scandal" is a "valid argument" against voting for Senator Brown. You can't make a decision based on that fact alone, but you can include it in a broader decision-making process.

The problem with the second definition is that it makes "Slavery increases cotton production" a valid argument for slavery, which invites confusion. I'd rather say that the statement about cotton production is a "good point" (even better: "truthful point") and then call the cost-benefit analysis where you eventually decide "increased cotton production isn't worth the suffering, and therefore slavery is wrong" a "valid argument".

I can't really tell from the original post in which way Eliezer is using "valid argument". I assumed the first way, because he uses the phrase "valid form of argument" a few times. But re-reading the post, maybe I was premature. But here's my opinion:

Fiction isn't the first type of valid argument because there are no stated premises, no stated conclusion, and no formal structure. Or, to put it another way, on what grounds could you claim that a work of fiction was an invalid argument?

Fiction can convincingly express the second type of valid argument (good point), and this is how I think of Uncle Tom's Cabin. "Slavery is bad because slaves suffer" is a good point against slavery, and Uncle Tom's Cabin is just a very emotionally intense way of making this point that is more useful than simple assertion would be for all the reasons previously mentioned.

My complaint in my original post is that fiction tends to focus the mind on a single good point with such emotional intensity that it can completely skew the rest of the cost-benefit analysis. For example, the hypothetical sweatshop book completely focuses the mind on the good point that people can suffer terribly while working in a sweatshop. Anyone who reads the sweatshop book is in danger of having this one point become so salient that it makes a "valid argument" of the first, more formal type much more difficult.

comment by Vladimir_Nesov · 2009-02-11T00:48:22.000Z · LW(p) · GW(p)

Well, my point is that fiction isn't argument at all, it's a theme for musing on your own questions. It's sometimes useful to read even something you know to be wrong, on a theme interesting for you, written by a thoughtful author. You don't expect to move towards agreement, you know the stuff is wrong, but you can light sparks of your own insight off its pages. When given a mathematical proof, its correctness is for you to appraise. Fiction gives you your own thoughts, take them or leave them.

comment by TreeFrog · 2009-02-11T01:16:27.000Z · LW(p) · GW(p)

Michael Vassar: I have no idea who you are, but I'll proceed on the assumption that the "job" you mention is one of representing Eliezer and/or the other OvercomingBias authors in some sort of business capacity.

I don't know Max on a personal level. We've talked a few times on his board and he might be dimly aware of my existence, but I make no claims as to what he will do if contacted by Eliezer or an agent of Eliezer's.

Serious discussions of potential OvercomingBias projects/movies and whatnot should be sent to Max's assistant, Ian Claudius - ian.claudius(AT)gmail.com. The Rudius people are smart and good content creators (multiple book contracts, one soon to be hit movie and stuff I actually like).

comment by David_Ellis · 2009-02-11T03:17:29.000Z · LW(p) · GW(p)

Whatever the potential abuses, I think fiction has a valid role to play in dealing with philosophical questions.

One example that comes to mind: I've observed in several discussions on the problem of evil that theists tend to want to discuss the matter in the most abstract possible terms. It seems to be easier to swallow a theodicy when you don't have the unpleasant facts of extreme suffering vividly in mind.

In the case of the POE, fiction can serve to cut through rationalizations in a way argumentation alone can never hope to do.

comment by Kyle3 · 2009-02-14T13:09:45.000Z · LW(p) · GW(p)


Took a bit after reading your babyeater pieces to get my thoughts in order, but my general picture is that you're mis-representing the human condition, and that the whole of the story relies upon that misrepresentation. This is humanity you're talking about. While the readership of OB may match your profile decently, the human species, even extended into a moderately improved state does not hold the value-set you represent.

  1. Child-love/protection is (a) proximity-focused, (b) stronger than abstract value of reciprocity. Folks are much better at getting over the idea that other folks are suffering than you give credit for. Hence the trade-off between superhappies modifying humans and humans saving baby-eater kids is a no-brainer. Nuke the local nova. Or better, point at a fundamental incompatibility between Baby-eater and Human ideology...that the SH plan misunderstands.

  2. Us vs. Them is under-done as well. Ender's choice...where someone protects human species, even at the cost of the other species, and then is later vilified by self and others is much more likely. Nuke the local nova.

  3. As far as I can tell, the story says starlines are dense and unpredictable, not that there's no other path. So...nuking Huygens star is too big a risk/cost for the uncertainty that baby-eaters will find another path. Nuke local, or don't bother.

Overall, I enjoyed it.

comment by nolrai · 2009-05-03T19:42:28.000Z · LW(p) · GW(p)

I think I have a co-operation instinct that is pushing me towards the supper happy future.

It feels better, but is probably not what I would do In real life. or I am more different then others then I give credit for.

comment by lessdazed · 2011-08-02T06:42:37.790Z · LW(p) · GW(p)

Telling people about the logical fallacy of generalization from fictional evidence doesn't make it go away; people may just go on generalizing from the story as though they had actually seen it happen.

That danger can be minimized in an elegant way: by referring people to a story that shows one shouldn't generalize from fictional evidence.

The more likely they are to inappropriately take the messages of stories to heart, the more they will be to take to heart the message not to do that.

comment by orthonormal · 2011-08-02T18:10:36.181Z · LW(p) · GW(p)

Can someone fix the formatting on this article?

Replies from: Alicorn, Vladimir_Nesov
comment by Alicorn · 2011-08-02T18:38:05.989Z · LW(p) · GW(p)

I can't see what's causing the problem in either the WYSIWYG editor or the HTML one, both of which show spaces around the words that are cramped in the published version. I'm confused.

Replies from: JGWeissman, Swimmer963
comment by JGWeissman · 2011-08-02T18:50:41.326Z · LW(p) · GW(p)

It looks like the indented quoted sections are implemented with "div" elements, which I recall have somehow caused spacing problems around other tags in the same article. It would probably help to change those sections to use that LW standard quoting format.

comment by Swimmer963 · 2011-08-02T19:08:26.805Z · LW(p) · GW(p)

That's happened to me before. I just put in extra spaces, so that when it was published it appeared normal. Still don't know what caused it.

comment by Vladimir_Nesov · 2011-08-02T19:59:41.037Z · LW(p) · GW(p)

Fixed. (It's more reliable to PM me, since I don't read all comments; it just so happens that you are among the people whose comments I'm subscribed to.)

Replies from: Swimmer963
comment by Swimmer963 · 2011-08-02T20:19:51.996Z · LW(p) · GW(p)

You can subscribe to people's comments? That's...pretty nifty! How do you go about doing it?

Replies from: AdeleneDawner
comment by AdeleneDawner · 2011-08-02T20:24:38.383Z · LW(p) · GW(p)

Go to the person's user page, and click 'subscribe to RSS feed' in the sidebar. This works on most pages on LW, in fact.

comment by [deleted] · 2013-11-06T18:21:38.704Z · LW(p) · GW(p)

I actually did try to send a long message consisting only of


Reading that was quite effective in pushing me towards Torture.

(I'm not being sarcastic.)