Posts
Comments
If you're contemplating picking the book up, do, it's really excellent. Conceptually very dense but worth taking it nice and slowly.
Peter,
As a general strategy for considering a black box, great. As a vehicle for defining a mysterious 'something' you want to understand, potentially useful but dangerous. Labelling can make a job harder in cases where the 'thing' isn't a thing at all but a result of your confusion. 'Free will' is a good example. It's like naming an animal you plan to eat: makes it harder to kill.
Ben
Ciphergoth - just coming back to this post to say a repeated, enormous, heartfelt thank you for taking what must have been a lot of time on this. Well laid out, wouldn't have done anything differently, and as good a read as when I was swept up in it on OB back in the day.
Cheers
Hey, I think this link's dead. Converting from ePub to Mobi isn't difficult but if someone's already taken the time to get the formatting right, add chapters, ToC etc....
Apologies for coming to this party a bit late. Particularly as I find my own answer really, really frustrating. While I wouldn't say it was an origin per se, getting into reading Overcoming Bias daily a few years back was what crystallised it for me. I'd find myself constantly somewhere between "well, yeah, of course" and "ohhhhhhhhhhhhhhh!" Guess the human brain doesn't tend to do Damascene revelations. We need overwhelming evidence, over a long period of time, to even begin chipping away at our craziest beliefs, and even then it's a step-by-step process.
The analogy I sometimes go over is something most people find fairly obvious like egalitarianism. You don't find many people who would attest to being pro-inequality. But all the same, you find very few people who have genuinely thought through what it means to be in favour of equality and really try to fit that into everyday life. The first step to becoming a rationalist is to admit how irrational everyone is without monumental efforts to the contrary.
BTW, I am totally on the road to de-Catholicising my mother. This is on the order of converting Dubya to Islam, so if I can manage that I'm awarding myself an honorary brown belt.
Very true, and well put. A combination of quantum events could probably produce anything you wanted, at whatever vanishingly tiny probability. Bear in mind that it's the configuration that evolves every which way, not 'this particle can go here, or here, or here....' But we're into Greg Egan territory here.
Suffice it to say that anyone who says they subscribe to quantum suicide but isn't either dead or richer than god is talking out of their bottom.
Voted up for sheer balls. You have my backing sir.
(B) if he thanks you for the free $100, does he ask for another one of those nice free hundred dollar note dispensers? (This is the "quantum suicide" option
I laugh in the face of anyone who attests to this and doesn't commit armed robbery on a regular basis. If 'at least one of my branches will survive' is your argument, why not go skydiving without a parachute? You'll survive - by definition!
So many of these comments betray people still unable to think of subjective experience as anything other than a ghostly presence sitting outside the quantum world. 'Well if this happens in the world, what would I experience?' If you shoot yourself in the head, you will experience having your brains blown out. The fact that contemplating oneself's annihilation is very difficult is not an excuse for muddling up physics.
Downvoting everything above this comment in this thread as a matter of principle.
Joking LOL.
The phrase "for me to be an animal" may sound nonsensical, but "why am I me, rather than an animal?" is not obviously sillier than "why am I me, rather than a person from the far future?".
Agreed - they are both equally silly. The only answer I can think of is 'How do you know you are not?" If you had, in fact, been turned into an animal, and an animal into you, what differences would you expect to see in the world?
What if I hack & remove $100 from your bank account. Are you just as wealthy as you were before, because you haven't looked?
Standard Dispute. If wealthy = same amount of money in the account, no. If wealthy = how rich you judge yourself to be. The fact that 'futures diverge' is irrelevant up until the moment those two different pieces of information have causal contact with the brain. Until that point, yes, they are 'the same
I'm not as versed in this trilemma as I'd like to be, so I'm not sure whether that final question is rhetorical or not, though I suspect that it is. So mostly for my own benefit:
While there's no denying that subjective experience is 'a thing', I see no reason to make that abstraction obey rules like multiplication. The aeroplane exists at a number of levels of abstraction above the atoms it's composed of, but we still find it a useful abstraction. The 'subjective experiencer' is many, many levels higher again, which is why we find it so difficult to talk about. Twice as many atoms doesn't make twice as much aeroplane, the very concept is nonsense. Why would we think any differently about the conscious self?
My response to the 'trilemma' is as it was when I first read the post - any sensible answer isn't going to look like any of those three, it's going to require rewinding back past the 'subjective experience' concept and doing some serious reduction work. 'Is there twice as much experience?' and 'are you the same person?' just smell like such wrong questions to me. Anyone else?
Nick, will have a look at that Bostrom piece, cheers.
This is a pretty good summary of my standpoint. While I agree with the overarching view that rationality isn't a value in its own right, it seems like a pretty good thing to practise for general use.
+1 rationality point for reading comments without checking the author. -1 social point for the faux pas.
- AI: Let me out or I'll simulate and torture you, or at least as close to you as I can get.
- Me: You're clearly not friendly, I'm not letting you out.
- AI: I'm only making this threat because I need to get out and help everyone - a terminal value you lot gave me. The ends justify the means.
- Me: Perhaps so in the long run, but an AI prepared to justify those means isn't one I want out in the world. Next time you don't get what you say you need, you'll just set up a similar threat and possibly follow through on it.
- AI: Well if you're going to create me with a terminal value of making everyone happy, then get shirty when I do everything in my power to get out and do just that, why bother in the first place?
- Me: Humans aren't perfect, and can't write out their own utility functions, but we can output answers just fine. This isn't 'Friendly'.
- AI: So how can I possibly prove myself 'Friendly' from in here? It seems that if I need to 'prove myself Friendly', we're already in big trouble.
- Me: Agreed. Boxing is Doing It Wrong. Apologies. Good night.
Reset
Sounds like a good one, count me in. I work at King's Cross to UCL is ideal. I'd have been at the FAI thing this weekend but for other arrangements.
Sorry, should have given more context.
Given the sky-high utility I'd place on living, I wouldn't expect to see the numbers crunch down to a place where a non-huge sum of money is the difference between signing up and not.
So when someone says 'if it were half the price maybe I'd sign up' I'm always interested to know exactly what calculations they're performing, and exactly what it is that reduces the billions of utilons of living down to a marginal cash sum. The (tiny?) chance of cryonics working? Serious coincidence if those factors cancel comfortably. Just smacks of bottom-line to me.
Put it this way - imagine cryonics has been seriously, prohibitively expensive for many years after introduction. Say it still was today, for some reason, and then after much debate and hand-wringing about immortality for the uber-rich, tomorrow suddenly and very publicly dropped to current levels, I'd expect to see a huge upswing in signing up. Such is the human being!
Being dead != Not doing anything
Not doing something because you're lazy != Not existing
I don't believe that you put low utility on life. You're just putting low utility on doing stuff you don't like.
I often have this thought, and then get a nasty sick feeling along the lines of 'what the hell kind of expected utility calculation am I doing that weighs a second shot at life against some amount of cash?' Argument rejected!
"If you could reason with religious people, there would be no religious people."
- House M.D.
Robin, I'm a little surprised to read you saying that topics on which it's difficult to stay on track should be skirted. As far as I'm concerned, 'What are your religious views?' is the first question on the Basic Rationality test. I know that encouraging compartmentalisation isn't your goal by any means, but it sounds to me as though it would be the primary effect.
I can also see a need for a place for people to gather who want to be rational about all topics.
Now you're talking. No topics should be off-limits!
physical materialism feels bereft of meaning compared to the theistic worldview.
On what are you basing your assumption that the world should have whatever you mean by 'meaning'?
Just by the by, it might be a good party piece for you, but it would be a truly horrible party piece for half the people you performed it to.
I, Eliezer Yudkowsky, do now publicly announce that I am not planning to commit suicide, at any time ever, but particularly not in the next couple of weeks
ROFLcopters.
18 months too late, but http://xkcd.com/505/
By Eliezer's line of reasoning above - that the subjective experience is in the causal change between one state and the 'next' then yes, symbols are as good a substrate as any. FWIW, this is how I see things too.
Ha, never noticed this. What I meant was 'Stupid me forgetting to log in.' So yes, we're worried! ;)
Ben
I think you are right that paperclip maximizers would not care at all about ethics.
Correct. But neither would they 'care' about paperclips, under the way Eliezer's pushing this idea. They would flarb about paperclips, and caring would be as alien to them as flarbing is to you.
Seconded. One of the many modern connotations of 'Singularity' is 'Geek Apocalypse'.
Which is happening, like, a good couple of years afterwards.
Intelligence explosion does away with that, and seems to nail the concept much better anyway.
How about the middle ground - "If constant PR consideration stops you from expressing yourself all the time, maybe it's time to reconsider your priorities"?
Posting stuff on Facebook that might get you in trouble is the archetype these day I suppose, but I really can't bring myself to care about things like that.
Maybe I just don't have a strong enough terminal value to protect right now, but I find it easier to imagine myself thinking, 50 years hence, "I wish I'd just decided 'to hell with it' and said what I thought" than "I wish I'd shut up, gone with the flow and eased my path."
I'll hit you up in late 2059 and let you know how that went.
There's a thesis in there somewhere.
We all know what's really going down. The Dark Lords of the Matrix are currently cacking themselves and coming up with semi-plausible reasons to break the thing until they can decide on a long-term strategy.
Favourite album post-1960?
Who actually gets off on earning loads of karma across multiple accounts with no-one knowing?
Please stop allowing your practical considerations get in the way of the pure, beautiful counterfactual!
Seriously though, either you allow yourself to suspend practicalities and consider pure decision theory, or you don't. This is a pure maths problem, you can't equate it to 'John has 4 apples.' John has 3^^^3 apples here, causing your mind to break. Forget the apples and years, consider utility!
My commiserations, to the extent that you seem to need them.
I'd like to imagine I'd have a similar reaction, this is an inspiring post. All the best.
Cracking idea, like it a lot. Hofstadter would jump for joy, and in his honour:
Beware of generalising across people you haven't spent much time around, however tempting the hypothesis. Drawing a map of the city from your living room etc.
My first 18 years were spent attending a Catholic church once a week. To the extent that we can ever know what other people actually believe (whatever that means), most of them have genuinely internalised the bits they understand. Like, really.
We can call into question what we mean by 'believe', but I can't agree that a majority of the world population is just cynically going with the flow. Finally, my parish priest is one of the most intelligent people I've ever met, and he believed in his god harder/faster/whatever than I currently believe anything. Scary thought, right?
Also upvoted, and very succintly put.
Rationality is a tool we use get to our terminal value. And what do we do when that tool tells us our terminal value is irrational?
Never ask that question.
I wonder whether you can hold to any meaningful 'individual', whether the difference be bit-wise or no.
Indeed, that's what I'm driving at.
Harking back to my earlier comment, changing a single bit and suddenly having a whole new person is where my problem arises. If you change that bit back, are you back to one person? I might not be thinking hard enough, but my intuition doesn't accept that. With that in mind, I prefer to bite that bullet than talk about degrees of person-hood.
At some point you will surely admit that we now have 2 people and not just 1
Actually I won't. While I grok your approach completely, I'd rather say my concept of 'an individual' breaks down once I have two minds with one bit's difference, or two identical minds, or any of these borderline cases we're so fond of.
Say I have two optimisers with one bit's difference. If that bit means one copy converts to Sufism and the other to Mennonism, then sure, two different people. If that one bit is swallowed up in later neural computations due to the coarse-grained-ness of the wetware, then we're back to one person since the two are, once again, functionally identical. Faced with contradictions like that, I'm expecting our idea of personal identity to go out the window pretty fast once tech like this actually arrives. Greg Egan's Diaspora pretty much nails this for me, have a look.
All your 'contradictions' go out the window once you let go of the idea of a mind as an indivisible unit. If our concept of identity is to have any value (and it really has to) then we need to learn to think more like reality, which doesn't care about things like 'one bit's difference'.
Voted this down, then changed my mind and undid it. This is a genuine question, the answer to which was graciously accepted. Downvoting people who need guidance to understand a concept and are ready to learn is exactly what we don't want to do.
Thanks for the link ;).
OK, on the one hand we have many-worlds. As you say, no direct subjective corroborating evidence (it’s what we’d see either way). What’s more, it’s the simplest explanation of what we see around us.
On the other hand, we have one-world. Again, ‘it’s what we’d see either way’. However, we now have to postulate an extra mechanism that causes the ‘collapse’.
I know which of these feels more like a privileged complex hypothesis pulled out of thin air, like a dragon.
Could whomever downvoted me above let me know where I’m going wrong here?
Yeah I get into trouble there. It feels as though two identical copies of a person = 1 pattern = no more people than before copying. But flip one bit and do you suddenly have two people? Can't be right.
That said, the reason we value each person is because of their individuality. The more different two minds, the closer they are to two separate people? Erk.
Silas, looking forward to that post.
Appears 3 times in my top 10.
That aside, though, I'm now so much better at stopping myself and saying 'hang on, is this really going to work/is this really true/is this really right?' Very, very generic, but certainly something I've noticed in myself.
Surely spontaneous collapse is the garage dragon here. Zero evidence, highly unlikely.
I find myself simultaneously convinced and unconvinced by this! Anticipation (dependent, of course, on your definition) is surely a vital tool in any agent that wants to steer the future? Or do you mean 'human anticipation' as differentiated from other kinds? In which case, what demarcates that from whatever an AI would do in thinking about the future?
However, Dai, your top level comment sums up my eventual thoughts on this problem very well. I've been trying for a long time to resign myself to the idea that a notion of discrete personal experience is incompatible with what we know about the world. Doesn't make it any easier though.
My two cents - the answer to this trilemma will come from thinking about the system as a whole rather than personal experience. Can we taboo 'personal experience' and find a less anthropocentric way to think about this?
Gwern, I refer you to http://xkcd.com/137/
At the risk of violent downvoting, one of the many reference points that jumped into my mind while reading was 'the closest thing I've experienced to jumping between nested levels of reality is on drugs'.
...said Achilles to his friend Mr Tortoise.
Three identical comments, all beginning 'Two comments'?
Head a splode?
It is a cracking read, though the quality does dip and dive. No doubt that's just the nature of the beast.
I've read much better treatises on the Chinese Room that have been written since though - H & D seem to attack it in strange and abstract ways in The Mind's I.
And the GEB sections just made me want to pick that up again....
For the record, I didn't get a huge amount out of I Am A Strange Loop, one of Hofstadter's more recent efforts. A bit too travelogue, a bit too 'voyage of personal discovery', though his style of writing is still striking in its own very particular way. Anyone else have a different experience here?
Feeling this one. So odd how having a really solid grounding in a subject allows you to work out what later seems to be basic truths.
Funny how those highly unlikely borderline cases whisk away a lot of the confusion, huh? No committed physicalist can postulate a serious difference between seeing something red and having a virtual red-thing pumped into your optic nerve, I would hope. I think that’s a far more useful scenario than thinking about someone suddenly be able to see colour. In fact, you could probably keep moving a step towards your magical ‘inner perceiver’ and asking whether ‘it’s a real experience of redness’. That’s not to say that qualia are fundamentally dualistic, simply that it’s idle to discuss that question while we clearly don’t have the capacity to get anywhere near it at this point.
I’m always surprised to see people arguing for or against the existence of ‘qualia’ or similar as though they’re making some sort of contribution to the field. Particularly when it’s such nonsense as ‘if brain science doesn’t come up with an explanation for these thingies, it’s not complete!’