Polarized gamma rays and manifest infinity 2011-07-30T06:56:49.555Z
Rational entertainment industry? 2010-12-10T15:55:48.002Z
Compatibilism in action 2010-11-23T17:58:04.506Z
The Curve of Capability 2010-11-04T20:22:48.876Z
The I-Less Eye 2010-03-28T18:13:13.358Z
Two probabilities 2010-02-15T14:18:11.036Z
Privileged Snuff 2010-01-22T05:38:36.259Z
Why safety is not safe 2009-06-14T05:20:55.442Z
Fire and Motion 2009-04-29T16:06:11.967Z


Comment by rwallace on Slowing Moore's Law: Why You Might Want To and How You Would Do It · 2012-03-13T00:59:37.564Z · LW · GW

This used to be an interesting site for discussing rationality. It was bad enough when certain parties started spamming the discussion channel with woo-woo about the machine Rapture, but now we have a post openly advocating terrorism, and instead of being downvoted to oblivion, it becomes one of the most highly upvoted discussion posts, with a string of approving comments?

I think I'll stick to hanging out on sites where the standard of rationality is a little better. Ciao, folks.

Comment by rwallace on Open Thread, February 15-29, 2012 · 2012-02-17T02:31:02.160Z · LW · GW

Example: Most people would save a young child instead of an old person if forced to choose, and it is not not just because the baby has more years left, part of the reason is because it seems unfair for the young child to die sooner than the old person.

As far as I'm concerned it is just because the baby has more years left. If I had to choose between a healthy old person with several expected years of happy and productive life left, versus a child who was terminally ill and going to die in a year regardless, I'd save the old person. It is unfair that an innocent person should ever have to die, and unfairness is not diminished merely by afflicting everyone equally.

Comment by rwallace on Diseased disciplines: the strange case of the inverted chart · 2012-02-07T23:09:32.308Z · LW · GW

That would be cheap and simple, but wouldn't give a meaningful answer for high-cost bugs, which don't manifest in such small projects. Furthermore, with only eight people total, individual ability differences would overwhelmingly dominate all the other factors.

Comment by rwallace on Scott Sumner on Utility vs Happiness [Link] · 2012-02-07T21:09:12.055Z · LW · GW

Sorry, I have long forgotten the relevant links.

Comment by rwallace on Diseased disciplines: the strange case of the inverted chart · 2012-02-07T21:08:36.070Z · LW · GW

We know that late detection is sometimes much more expensive, simply because depending on the domain, some bugs can do harm (letting bad data into the database, making your customers' credit card numbers accessible to the Russian Mafia, delivering a satellite to the bottom of the Atlantic instead of into orbit) much more expensive than the cost of fixing the code itself. So it's clear that on average, cost does increase with time of detection. But are those high-profile disasters part of a smooth graph, or is it a step function where the cost of fixing the code typically doesn't increase very much, but once bugs slip past final QA all the way into production, there is suddenly the opportunity for expensive harm to be done?

In my experience, the truth is closer to the latter than the former, so that instead of constantly pushing for everything to be done as early as possible, we would be better off focusing our efforts on e.g. better automatic verification to make sure potentially costly bugs are caught no later than final QA.

But obviously there is no easy way to measure this, particularly since the profile varies greatly across domains.

Comment by rwallace on Diseased disciplines: the strange case of the inverted chart · 2012-02-05T17:02:38.541Z · LW · GW

Because you couldn't. In the ancestral environment, there weren't any scientific journals where you could look up the original research. The only sources of knowledge were what you personally saw and what somebody told you. In the latter case, the informant could be bullshitting, but saying so might make enemies, so the optimal strategy would be to profess belief in what people told you unless they were already declared enemies, but base your actions primarily on your own experience; which is roughly what people actually do.

Comment by rwallace on Open Thread, February 1-14, 2012 · 2012-02-01T12:35:14.492Z · LW · GW

That's not many worlds, that's quantum immortality. It's true that the latter depends on the former (or would if there weren't other big-world theories, cf. Tegmark), but one can subscribe to the former and still think the latter is just a form of confusion.

Comment by rwallace on I've had it with those dark rumours about our culture rigorously suppressing opinions · 2012-01-26T20:28:53.134Z · LW · GW

True. The usual reply to that is "we need to reward the creators of information the same way we reward the creators of physical objects," and that was the position I had accepted until recently realizing, certainly we need to reward the creators of information, but not the same way - by the same kind of mechanism - that we reward the creators of physical objects. (Probably not by coincidence, I grew up during the time of shrink-wrapped software, and only re-examined my position on this matter after that time had passed.)

Comment by rwallace on I've had it with those dark rumours about our culture rigorously suppressing opinions · 2012-01-26T20:23:40.058Z · LW · GW

To take my own field as an example, as one author remarked, "software is a service industry under the persistent delusion that it is a manufacturing industry." In truth, most software has always been paid for by people who had reason other than projected sale of licenses to want it to exist, but this was obscured for a couple of decades by shrinkwrap software, shipped on floppy disks or CDs, being the only part of the industry visible to the typical nonspecialist. But the age of shrinkwrap software is passing - outside entertainment, how often does the typical customer buy a program these days? - yet the industry is doing fine. We just don't need copyright law the way we thought we did.

Comment by rwallace on I've had it with those dark rumours about our culture rigorously suppressing opinions · 2012-01-26T20:14:44.139Z · LW · GW

We can't. We can only sensibly define them in the physical universe which is based on matter, with its limitations of "only in one place at a time" and "wears out with use" that make exclusive ownership necessary in the first place. If we ever find a way to transcend the limits of matter, we can happily discard the notion of property altogether.

Comment by rwallace on I've had it with those dark rumours about our culture rigorously suppressing opinions · 2012-01-26T20:10:56.469Z · LW · GW

I took the post to be asking for opinions sufficiently far outside the mainstream to be rarely discussed even here, and I haven't seen a significant amount discussion of this one. Then again, that could be because I wasn't particularly looking; I used to be of the opinion "intellectual property law has gone too far and needs to be cut back, but of course we can't do away with it entirely," and only recently looked more closely at the but of course part and realized it didn't hold water. If this opinion is more common than I had given it credit for, great!

Comment by rwallace on I've had it with those dark rumours about our culture rigorously suppressing opinions · 2012-01-26T20:07:39.345Z · LW · GW

Sure. My answer is no, it does not.

Comment by rwallace on I've had it with those dark rumours about our culture rigorously suppressing opinions · 2012-01-26T00:08:47.392Z · LW · GW

Not only is intellectual property law in its current form destructive, but the entire concept of intellectual property is fundamentally wrong. Creating an X does not give the creator the right to point a gun at everyone else in the universe who tries to arrange matter under their control into something similar to X. In programming terminology, property law should use reference semantics, not value semantics. Of course it is true that society needs to reward people who do intellectual work, just as much as people who do physical work, but there are better justified and less harmful ways to accomplish this than intellectual property law.

Comment by rwallace on POSITION: Design and Write Rationality Curriculum · 2012-01-20T19:54:12.312Z · LW · GW

Ill posed does not necessarily mean impossible. Most of the problems we deal with in real life are ill posed, but we still usually manage to come up with solutions that are good enough for the particular contexts at hand. What it does mean is that we shouldn't expect the problem in question to be definitely solved once and for all. I'm not arguing against attempting to test rationality. I'm arguing against the position some posters have taken that there's no point even trying to make progress on rationality until the problem of testing it has been definitely solved.

Comment by rwallace on POSITION: Design and Write Rationality Curriculum · 2012-01-20T19:36:13.102Z · LW · GW

But doesn't it seem that if you decompartmentalized with correct beliefs you should do way better?

Maybe; there are all sorts of caveats to that. But that aside, more directly on the question of tests:

Possibly in a testable way?

You still run into the problem that the outcome depends greatly on context and phrasing. There is the question with turning over cards to test a hypothesis, on which people's performance dramatically improves when you rephrase it as an isomorphic question about social rules. There are the trolley questions and the specks versus torture question and the ninety-seven percent versus one hundred percent question, on which the right answer depends entirely on whether you treat it as a mathematical question that happens to be expressed in English syntax or a question about what you should do if you believed yourself to really be in that situation. There are questions about uncertain loss isomorphic to questions about uncertain gain where people nonetheless give different answers, which is irrational if considered as a material problem, but rational in the more likely and actual situation where the only thing at stake is social status, which sometimes does depend on how the question was phrased. Etc.

That's why I called the testing problem ill posed; it's not just that it's hard to figure out the solution, it's hard to see what would be the criteria of a good solution in the first place.

Comment by rwallace on POSITION: Design and Write Rationality Curriculum · 2012-01-20T18:37:24.671Z · LW · GW

Testing rationality is something of an ill posed problem, in part because the result depends greatly on context. People spout all kinds of nonsense in a social context where it's just words, but usually manage to compartmentalize the nonsense in a material context where they will be affected by the results of their actions. (This is a feature! Given that evolution wasn't able to come up with minds that infallibly distinguish true beliefs from false ones, it's good that at least it came up with a way to reduce the harm from false beliefs.) I'm not sure how to create an accurate test in the face of that.

Your martial arts analogy isn't a bad one. The outcome of a karate contest is often not the same as the outcome of a street fight between the same participants. There are any number of cases of a black belt karateka with ten years training getting into a fight with a scrawny untrained criminal, and getting his ass kicked in three seconds flat. Martial arts practitioners have had this testing problem for centuries and still don't seem close to solving it, which doesn't make for optimism about our prospects of solving the rationality testing problem this century. Given that, proceeding as best we can in the absence of a comprehensive and accurate test seems reasonable.

Comment by rwallace on [Meta] No LessWrong Blackout? · 2012-01-18T23:51:07.673Z · LW · GW

"The price of freedom is eternal vigilance."

It would be wonderful if defending freedom were a one-off job like proving Fermat's Last Theorem. As it turns out, it's an endlessly recurring job like fighting disease; unfortunate, but that's the way it is. And yes, sometimes our efforts fail, and freedoms are lost or people get sick and die. But the answer to that is to work harder and smarter, not to give up.

Comment by rwallace on The Gift I Give Tomorrow · 2012-01-11T14:10:49.264Z · LW · GW

Most of this post, along with the previous posts in the series, is both beautiful and true - the best combination. It's a pity it had to be mixed in with the meme about computers magically waking up with superpowers. I don't think that meme is necessary here, any more than it's necessary to believe the world was created in 4004 BC to appreciate Christmas. Taking it out - discussing it in separate posts if you wish to discuss it - is the major improvement I would suggest.

Comment by rwallace on A case study in fooling oneself · 2011-12-15T06:37:27.302Z · LW · GW

Good points, upvoted. But in fairness, I think the ink blot analogy is a decent one.

Imagine you asked the question about the ink blot to a philosopher in ancient Greece, how might he answer? He might say there is no definite number. Or he might say there must be some underlying reality, even though he doesn't know for sure what it is; and the best guess says it's based on atoms; so he might reply that he doesn't know the answer, but hopefully it might be possible in principle to calculate it if you could count atoms.

I think that's about where we are regarding the Born probabilities and number or measure of different worlds in MWI right now.

Comment by rwallace on [SEQ RERUN] Stop Voting For Nincompoops · 2011-12-13T07:26:28.405Z · LW · GW

There is a wonderfully evocative term, Stand Alone Complex, from the anime series of the same name, which refers to actions taken by people behaving as though they were part of a conspiracy even though no actual conspiracy is present. It's pretty much tailor-made for this case.

Mencius Moldbug calls this instance the Cathedral, in an insightful series of articles indexed here.

Comment by rwallace on Humans Shouldn't make Themselves Smarter? · 2011-12-12T09:47:21.234Z · LW · GW

You could also trade off things that were more important in the ancestral environment than they are now. For example, social status (to which the neurotypical brain devotes much of its resources) is no longer the evolutionary advantage that it used to be.

Comment by rwallace on Richard Carrier on the Singularity · 2011-12-05T15:56:07.087Z · LW · GW

Only if you take 'ten times smarter' to mean multiplying IQ score by ten. But since the mapping of the bell curve to numbers is arbitrary in the first place, that's not a meaningful operation; it's essentially a type error. The obvious interpretation of 'ten times smarter' within the domain of humans is by percentile, e.g. if the author is at the 99% mark, then it would refer to the 99.9% mark.

And given that, his statement is true; it is a curious fact that IQ has diminishing returns, that is, being somewhat above average confers significant advantage in many domains, but being far above average seems to confer little or no additional advantage. (My guess at the explanation: first, beyond a certain point you have to start making trade-offs from areas of brain function that IQ doesn't measure; second, Amdahl's law.)

Comment by rwallace on "Ray Kurzweil and Uploading: Just Say No!", Nick Agar · 2011-12-03T17:46:05.474Z · LW · GW

There is kidnapping for interrogation, slavery and torture today, so there is no reason to believe there won't be such in the future. But I don't believe it will make sense in the future to commit suicide at the mere thought, any more than it does today.

As for whether such a society will exist, I think it's possible it may. It's possible there may come a day when people don't have to die. And there is a better chance of that happening if we refrain from poisoning our minds with scare stories optimized for appeal to primate brains over correspondence to external reality.

Comment by rwallace on "Ray Kurzweil and Uploading: Just Say No!", Nick Agar · 2011-12-03T13:59:32.830Z · LW · GW

I've been snarky for this entire conversation - I find advocacy of death extremely irritating - but I am not just snarky by any means. The laws of physics as now understood allow no such thing, and even the author of the document to which you refer - a master of wishful thinking - now regards it as obsolete and wrong. And the point still holds - you cannot benefit today the way you could in a post-em world. If you're prepared to throw away billions of years of life as a precaution against the possibility of billions of years of torture, you should be prepared to throw away decades of life as a precaution against the possibility of decades of torture. If you aren't prepared to do the latter, you should reconsider the former.

Comment by rwallace on "Ray Kurzweil and Uploading: Just Say No!", Nick Agar · 2011-12-03T13:53:40.275Z · LW · GW

An upload, at least of the early generations, is going to require a supercomputer the size of a rather large building to run, to point out just one of the reasons why the analogy with playing a pirate MP3 is entirely spurious.

Comment by rwallace on Rationality Quotes December 2011 · 2011-12-03T13:50:07.728Z · LW · GW

Warhammer 40K is one of those settings that is highly is open to interpretation. My interpretation is that it's in a situation where things could be better and could be worse, victory and defeat are both very much on the cards, and hope guided by cold realism is one of the main factors that might tip the balance towards the first outcome. I consider it similar in that regard to the Cthulhu mythos, and for that matter to real life.

Comment by rwallace on "Ray Kurzweil and Uploading: Just Say No!", Nick Agar · 2011-12-03T08:08:05.018Z · LW · GW

If you postulate ems that can run a million subjective years a minute (which is not at all scientifically plausible), the mainline copies can do that as well, which means talking about wall clock time at all is misleading; the new subjective timescale is the appropriate one to use across the board.

As for the rest, people are just as greedy today as they will be in the future. Organized criminals could torture you until you agree to sign over your property to them. Your girlfriend could pour petrol over you and set you on fire while you're asleep. If you sign up for a delivery or service with Apple and give them your home address, you're trusting them not to send thugs around to your house and kidnap you. Ever fly on an airliner? Very few, perhaps no one, will have the engineering skill to fly without someone else's assistance. When you're on the plane, you're trusting the airline not to deliver you to a torture camp. Is anyone worthy of that trust? And even if you get home safely, how will you stay safe while you're asleep? And how will you protect yourself against criminals?

Does committing suicide today sound a more plausible idea now?

Comment by rwallace on "Ray Kurzweil and Uploading: Just Say No!", Nick Agar · 2011-12-03T07:37:44.794Z · LW · GW

The comment holds regardless. In today's world, you can only be tortured for a few decades, but by the same token you can only lose a few decades of lifespan by committing suicide. If in some future world you can be tortured for a billion years, then you will also be losing a billion years of happy healthy life by committing suicide. If you think the mere possibility of torture - with no evidence that it is at all likely - will be grounds for committing suicide in that future world, then you should think it equally good grounds for committing suicide today. If you agree with me that would be insanely irrational today, you should also agree it will be insanely irrational in that future world.

Comment by rwallace on "Ray Kurzweil and Uploading: Just Say No!", Nick Agar · 2011-12-03T07:25:17.032Z · LW · GW

Also, in the absence of any evidence that this is at all unlikely to occur.

If you think the situation is that symmetrical, you should be indifferent on the question of whether to commit suicide today.

But notice the original poster does not dwell on the probability of this scenario, only on its mere possibility.

If it had been generated as part of an exhaustive listing of all possible scenarios, I would have refrained from comment. As it is, being raised in the context of a discussion on whether one should try for uploading in the unlikely event one lives that long, it's obviously intended to be an argument for a negative answer, which means it constitutes:


  2. Advocacy of death.

Comment by rwallace on "Ray Kurzweil and Uploading: Just Say No!", Nick Agar · 2011-12-03T07:10:39.117Z · LW · GW

With the possibility? Of course not. Anything that doesn't involve a logical self-contradiction is possible. My disagreement is with the idea that it is sane or rational to base decisions on fantasies about being kidnapped and tortured in the absence of any evidence that this is at all likely to occur.

Comment by rwallace on "Ray Kurzweil and Uploading: Just Say No!", Nick Agar · 2011-12-03T06:50:27.916Z · LW · GW

If you think that kind of argument holds water, you should commit suicide today lest a sadist kidnap you and torture you in real life.

Comment by rwallace on Open Thread: December 2011 · 2011-12-02T13:58:30.858Z · LW · GW

No. The mainstream expectation has pretty much always been that locations conducive to life would be reasonably common; the results of the last couple of decades don't overturn the expectation, they reinforce it with hard data. The controversy has always been on the biological side: whether going from the proverbial warm little pond to a technological civilization is probable (in which case much of the Great Filter must be in front of us) or improbable (in which case we can't say anything about what's in front of us one way or the other). For what it's worth, I think the evidence is decisively in favor of the latter view.

Comment by rwallace on Life Extension versus Replacement · 2011-11-30T11:50:13.758Z · LW · GW

I'm perfectly prepared to bite this bullet. Extending the life of an existing person a hundred years and creating a new person who will live for a hundred years are both good deeds, they create approximately equal amounts of utility and I believe we should try to do both.

Comment by rwallace on Connecting Your Beliefs (a call for help) · 2011-11-22T00:00:41.831Z · LW · GW

Thanks for the link, yes, that does seem to be a different opinion (and some very interesting posts).

I agree with you about the publishing and music industries. I consider current rampant abuse of intellectual property law to be a bigger threat than the Singularity meme, sufficiently so that if your comparative advantage is in politics, opposing that abuse probably has the highest expected utility of anything you could be doing.

Comment by rwallace on Connecting Your Beliefs (a call for help) · 2011-11-21T10:16:55.092Z · LW · GW

That's awfully vague. "Whatever window of time we had", what does that mean?

The current state of the world is unusually conducive to technological progress. We don't know how long this state of affairs will last. Maybe a long time, maybe a short time. To fail to make progress as rapidly as we can is to gamble the entire future of intelligent life on it lasting a long time, without evidence that it will do so. I don't think that's a good gamble.

There's one kind of "technological progress" that SIAI opposes as far as I can tell: working on AGI without an explicit focus on Friendliness.

I have seen claims to the contrary from a number of people, from Eliezer himself a number of years ago up to another reply to your comment right now. If SIAI were to officially endorse the position you just suggested, my assessment of their expected utility would significantly increase.

Comment by rwallace on Connecting Your Beliefs (a call for help) · 2011-11-21T00:58:09.556Z · LW · GW

Or human communications may stop improving because they are good enough to no longer be a major bottleneck, in which case it may not greatly matter whether other possible minds could do better. Amdahl's law: if something was already only ten percent of total cost, improving it by a factor of infinity would reduce total cost by only that ten percent.

Comment by rwallace on Connecting Your Beliefs (a call for help) · 2011-11-20T21:44:37.323Z · LW · GW

We've had various kinds of Luddism before, but this one is particularly lethal in being a form that appeals to people who had been technophiles. If it spreads enough, best case scenario is the pool of people willing to work on real technological progress shrinks, worst case scenario is regulation that snuffs out progress entirely, and we get to sit around bickering about primate politics until whatever window of time we had runs out.

Comment by rwallace on Connecting Your Beliefs (a call for help) · 2011-11-20T21:36:58.828Z · LW · GW

Well, any sequence of events can be placed in a narrative frame with enough of a stretch, but the fact remains that different sequence of events differ in their amenability to this; fiction is not a random sampling from the space of possible things we could imagine happening, and the Singularity is narratively far stronger than most imaginable futures, to a degree that indicates bias we should correct for. I've seen a fair bit of strong Singularity fiction at this stage, though being, well, singular, it tends not to be amenable to repeated stories by the same author the way Heinlein's vision of nuclear powered space colonization was.

Comment by rwallace on Connecting Your Beliefs (a call for help) · 2011-11-20T20:25:51.875Z · LW · GW

We should update away from beliefs that the future will resemble a story, particularly a story whose primary danger will be fought by superheroes (most particularly for those of us who would personally be among the superheroes!) and towards beliefs that the future will resemble the past and the primary dangers will be drearily mundane.

Comment by rwallace on Connecting Your Beliefs (a call for help) · 2011-11-20T14:53:23.960Z · LW · GW

Okay, to look at some of the specifics:

Superior processing power. Evidence against would be the human brain already being close to the physical limits of what is possible.

The linked article is amusing but misleading; the described 'ultimate laptop' would essentially be a nuclear explosion. The relevant physical limit is ln(2)kT energy dissipated per bit erased; in SI units at room temperature this is about 4e-21. We don't know exactly how much computation the human brain performs; middle-of-the-road estimates put it in the ballpark of 1e18 several-bit operations per second for 20 watts, which is not very many orders of magnitude short of even the theoretical limit imposed by thermodynamics, let alone whatever practical limits may arise once we take into account issues like error correction, communication latency and bandwidth, and the need for reprogrammability.

Superior serial power: Evidence against would be an inability to increase the serial power of computers anymore.

Indeed we hit this some years ago. Of course as you observe, it is impossible to prove serial speed won't start increasing again in the future; that's inherent in the problem of proving a negative. If such proof is required, then no sequence of observations whatsoever could possibly count as evidence against the Singularity.

Superior parallel power:

Of course uses can always be found for more parallel power. That's why we humans make use of it all the time, both by assigning multiple humans to a task, and increasingly by placing multiple CPU cores at the disposal of individual humans.

Improved algorithms:

Finding these is (assuming P!=NP) intrinsically difficult; humans and computers can both do it, but neither will ever be able to do it easily.

Designing new mental modules:

As for improved algorithms.

Modifiable motivation systems:

An advantage when they reduce akrasia, a disadvantage when they make you more vulnerable to wireheading.

Copyability: Evidence against would be evidence that minds cannot be effectively copied, maybe because there won't be enough computing power to run many copies.

Indeed there won't, at least initially; supercomputers don't grow on trees. Of course, computing power tends to become cheaper over time, but that does take time, so no support for hard takeoff here.

Alternatively, that copying minds would result in rapidly declining marginal returns and that the various copying advantages discussed by e.g. Hanson and Shulman aren't as big as they seem.

Matt Mahoney argues that this will indeed happen because an irreducible fraction of the knowledge of how to do a job is specific to that job.

Perfect co-operation:

Some of the more interesting AI work has been on using a virtual market economy to allocate resources between different modules within an AI program, which suggests computers and humans will be on the same playing field.

Superior communication:

Empirically, progress in communication technology between humans outpaces progress in AI, and has done so for as long as digital computers have existed.

Transfer of skills:

Addressed under copyability.

Various biases:

Hard to say, both because it's very hard to see our own biases, and because a bias that's adaptive in one situation may be maladaptive in another. But if we believe maladaptive biases run deep, such that we cannot shake them off with any confidence, then we should be all the more skeptical of our far beliefs, which are the most susceptible to bias.

Of course, there is also the fact that humans can and do tap the advantages of digital computers, both by running software on them, and in the long run potentially by uploading to digital substrate.

Comment by rwallace on Connecting Your Beliefs (a call for help) · 2011-11-20T10:00:07.935Z · LW · GW

I discuss some of it at length here:

I'll also ask the converse question: given that you can't typically prove a negative (I can't prove the nonexistence of psychic powers or flying saucers either), if what we are observing doesn't constitute evidence against the Singularity in your opinion, then what would?

Comment by rwallace on Connecting Your Beliefs (a call for help) · 2011-11-20T07:57:33.402Z · LW · GW

I understand perfectly well how a hypothetical perfectly logical system would work (leaving aside issues of computational tractability etc.). But then, such a hypothetical perfectly logical system wouldn't entertain such far mode beliefs in the first place. What I'm discussing is the human mind, and the failure modes it actually exhibits.

Comment by rwallace on Connecting Your Beliefs (a call for help) · 2011-11-20T07:40:42.433Z · LW · GW

So your suggestion is that we should de-compartmentalize, but in the reverse direction to that suggested by the OP, i.e. instead of propagating forward from ridiculous far beliefs, become better at back-propagating and deleting same? There is certainly merit in that suggestion if it can be accomplished. Any thoughts on how?

Comment by rwallace on Connecting Your Beliefs (a call for help) · 2011-11-20T07:10:15.927Z · LW · GW

That's actually a good question. Let me rephrase it to something hopefully clearer:

Compartmentalization is an essential safety mechanism in the human mind; it prevents erroneous far mode beliefs (which we all adopt from time to time) from having disastrous consequences. A man believes he'll go to heaven when he dies. Suicide is prohibited in a patch for the obvious problem, but there's no requirement to make an all-out proactive effort to stay alive. Yet when he gets pneumonia, he gets a prescription for penicillin. Compartmentalization literally saves his life. In some cases many other lives, as we saw when it failed on 9/11.

Here we have a case study where a man of intelligence and goodwill redirected his entire life down a path of negative utility on the basis of reading a single paragraph of sloppy wishful thinking backed up by no evidence whatsoever. (The most straightforward refutation of that paragraph is that creating a machine with even a noteworthy fraction of human intelligence is far beyond the capacity of any human mind; the relevant comparison of such a machine if built would be with that which created it, which would have to be a symbiosis of humanity and its technology as a whole - with that symbiosis necessarily being much more advanced than anything we have today.) What went wrong?

The most obvious part of the answer is that this is an error to which we geeks are particularly prone. (Supporting data: terrorists are disproportionately likely to be trained in some branch of engineering.) Why? Well, we are used to dealing in domains where we can actually apply long chains of logic with success; particularly in the age range when we are old enough to have forgotten how fallible were our first attempts at such logic, yet young enough to be still optimists, it's an obvious trap to fall into.

Yet most geeks do actually manage to stay out of the trap. What else goes wrong?

It seems to me that there must be a parameter in the human mind for grasping the inertia of the world, for understanding at a gut level how much easier is concept than reality, that we can think in five minutes of ideas that the labor of a million people for a thousand years cannot realize. I suppose in some individuals this parameter must be turned up too high, and they fall too easily into the trap of learned helplessness. And in some it must be turned too low, and those of us for whom this is the case undertake wild projects with little chance of success; and if ninety-nine fail for every one who succeeds, that can yet drive the ratchet of progress.

But we easily forget that progress is not really a ratchet, and the more advanced our communications, the more lethal bad ideas become, for just as our transport networks spread disease like the 1918 flu epidemic which killed more people in a single year than the First World War killed in four years, so our communication networks spread parasite memes deadlier still. And we can't shut down the networks. We need them too badly.

I've seen the Singularity mutate from a harmless, even inspiring fantasy, to a parasite meme that I suspect could well snuff out the entire future of intelligent life. It's proving itself in many cases immune to any weight of evidence against it; perhaps worst of all, it bypasses ethical defenses, for it can be spread by people of honest goodwill.

Compartmentalization seems to be the primary remaining defense. When that fails, what have we left? This is not a rhetorical question; it may be one of the most important in the world right now.

Comment by rwallace on Babyeater's dilemma · 2011-11-16T03:15:03.831Z · LW · GW

You are worried that, given your assumptions, civilizations might not be willing to pay an extremely high price to do things that aliens would like if they knew about them, which they don't.

But one of your assumptions is that every civilization has a moral system that advocates attacking and enslaving everyone they meet who thinks differently from them.

It would be worrying if a slightly bad assumption led to a very bad conclusion, but a very bad assumption leading to a slightly bad conclusion doesn't strike me as particularly problematic.

Comment by rwallace on Why would an AI try to figure out its goals? · 2011-11-10T21:24:37.564Z · LW · GW

Well yes. You give this list of things you claim are universal instrumental values, and it sounds like a plausible idea in our heads, but when we look at the real world, we find humans and other agents tend not in fact possess these, even as instrumental values.

Comment by rwallace on Why would an AI try to figure out its goals? · 2011-11-10T17:58:35.168Z · LW · GW

In the case of chess programs, the argument is simply false. Chess programs do not in fact exhibit anything remotely resembling the described behavior, nor would they do so even if given infinite computing power. This despite the fact that they exhibit extremely high performance (playing chess better than any human) and do indeed have a simple goal.

Comment by rwallace on Why would an AI try to figure out its goals? · 2011-11-10T17:10:08.642Z · LW · GW

It is more that a wide range of simple goals gives rise to a closely-related class of behaviours

But that is only true by a definition of 'simple goals' under which humans and other entities that actually exist do not have simple goals. You can have a theory that explains the behavior that occurs in the real world, or you can have a theory that admits Omohundro's argument, but they are different theories and you can't use both in the same argument.

Comment by rwallace on Why would an AI try to figure out its goals? · 2011-11-10T16:44:56.462Z · LW · GW

A mind very like humans' that had the ability to test out new brain components and organizations seems like it would fit it.

Not really, because as you say, there are many constraints preventing it from being applicable, of which difficulty changing our brains' design is just one, so with that constraint removed, the argument would still not be applicable.

Comment by rwallace on Why would an AI try to figure out its goals? · 2011-11-10T16:32:26.098Z · LW · GW

Suppose we grant all this. Very well, then consider what conclusions we can draw from it about the behavior of the hypothetical AI originally under discussion. Clearly no matter what sequence of actions the AI were to carry out, we would be able to explain it with this theory. But a theory that can explain any observations whatsoever, makes no predictions. Therefore, contrary to Omohundro, the theory of optimization does not make any predictions about the behavior of an AI in the absence of specific knowledge of the goals thereof.