Posts
Comments
Keep in mind that people who apply serious life-changing ideas after reading about them in fiction are the exception rather than the norm. Most people who aren't exceptionally intellect-oriented need to personally encounter someone who "has something" that they themselves wish they had, and then have some reason to think that they can imitate them in that respect. Fiction just isn't it, except possibly in some indirect ways. Rationalist communities competing in the "real-world" arena of people living lives that other people want to and can emulate are a radically more effective angle for people who don't identify strongly with their intellectual characteristics.
It seems at best fairly confused to say that an L-zombie is wrong because of something it would do if it were run, simply because we evaluated what it would say or do against the situation where it didn't. Where you keep saying "is" and "concludes" and "being" you should be saying "would", "would conclude", and "would be", all of which is a gloss for "would X if it were run", and in the (counter-factual) world where the L-zombie "would" do those things it "would be running" and therefore "would be right". Being careful with your tenses here will go a long way.
Nonetheless I think the concept of an L-zombie is useful, if only to point out that computation matters. I can write a simple program that encapsulates all possible L-zombies (or rather would express them all, if it were run), yet we wouldn't consider that program to be those consciousnesses--a point well worth remembering in numerous examinations of the topic.
Once you know about affective death spirals, you can use them in tricky ways. Consider for example, that you got into an affective death spiral about capital "R" Rationality which caused you to start entertaining false delusions (like that teaching Rationality to your evil stepmother would finally make her love you, or whatever). If you know that this is an affective death spiral, you can do an "affective death spiral transfer" that helps you avoid the negative outcome without needing to go to war with your own positive feelings: in this case, realise that it's incredibly awesome that Rationality is so cool that it can even help you correct an affective death spiral about itself. Of course, you have to be careful to become actually good at this (but you get Rationality points for realising this, and triple Rationality points for actually achieving it. Awesome!!!). You also get huge Rationality boosts for realising failure modes in your pursuit of Rationality in general (because that's totally Rational too! See how that works?).
Affective death spirals are like anti-akrasia engines, so getting rid of them entirely might be substantially less advantageous than applying some clever munchkinry to them.
Ah well, I had to ask. I know religion is usually the "other team" for us, so I hope I didn't push any buttons by asking--definitely not my intention.
This article is awesome! I've been doing this kind of stuff for years with regards to motivation, attitudes, and even religious belief. I've used the terminology of "virtualisation" to talk about my thought-processes/thought-rituals in carefully defined compartments that give me access to emotions, attitudes, skills, etc. I would otherwise find difficult. I even have a mental framework I call "metaphor ascendence" to convert false beliefs into virtualised compartments so that they can be carefully dismantled without loss of existing utility. It's been nearly impossible to explain to other people how I do and think about this, though often you can show them how to do it without explaining. And for me the major in-road was totally a realisation that there exist tasks which are only possible if you believe they are--guess I'll have to check out The Phantom Tollbooth (I've never read it).
This might be a bit of a personal question (feel free to pm or ignore), but have you by any chance done this with religious beliefs? I felt like I got a hint of that between the lines and it would be amazing to find someone else who does this. I've come across so many people in my life who threw away a lot of utility when they left religion, never realising how much of it they could keep or convert without sacrificing their integrity. One friend even teasingly calls me the "atheist Jesus" because of how much utility I pumped back into his life just by leveraging his personal religious past. Religion has been under strong selective pressure for a long time, and has accumulated a crapload of algorithmic optimisations that can easily get tossed by their apostates just because they're described in terms of false beliefs. My line is always, "I would never exterminate a nuisance species without first sequencing its DNA." You just have to remember that asking the organism about its own DNA is a silly strategy.
Anyways, I could go on for a long time about this, but this article has given me the language to set up a new series I've been trying to rework for Less Wrong, along the lines of this, so I better get cracking. But the buzz of finding someone like-minded is an awesome bonus. Thank you so much for posting.
p.s. I have to agree with various other commenters that I wouldn't use the "dark arts" description myself--mind optimisation is at the heart of legit rationality. But I see how it definitely makes for useful marketing language, so I won't give you too much of a hard time for it.
You seem to be making a mistake in treating bridge rules/hypotheses as necessary--perhaps to set up a later article?
I, like Cai, tend to frame my hypotheses in terms of a world-out-there model combined with bridging rules to my actual sense experience; but this is merely an optimisation strategy to take advantage of all my brain's dedicated hardware for modelling specific world components, preprocessing of senses, etc.. The bridging rules certainly aren't logically required. In practice there is an infinite family of equivalent models over my mental experience which would be totally indistinguishable, regardless of how I choose to "format" that idea mentally. My choice of mental model format is purely about efficiency considerations, not a claim about either my senses or the phenomena behind their behaviour. I'm just better at Tic tac toe than JAM.
To see this, let's say Cai uses python internally to describe zir hypotheses A and B in their entirety. Clearly, ze can write either program with or without bridging rules and still have it yield identical predictions in all possible circumstances. Cai's true hypothesis is the behaviour of the python program, regardless of how ze actually structures it: the combined interaction of all of it together. Both hypotheses could be written purely in terms of predictions on how Cai's senses will change, thereby eliminating the "type error" issue. And if Cai is as heavily optimised for a particular structure of hypothesis as humans are, Cai can just use that--but for performance reasons, not because Cai has some magical way of knowing at what level of abstraction zir existence is implemented. Alternatively Cai might use a particular hypothesis structure because of the programmer's arbitrary decision when writing zir. But the way the hypothesis is structured mentally isn't a claim about how the universe works. The "hard problem of consciousness" is a problem about human intuitions, not a math problem.
I’ve got kind of a fun rationalist origin story because I was raised in a hyper-religious setting and pretty much invented rationalism for use in proselytisation. This placed me on a path of great transformation in my own personal beliefs, but one that has never been marked by a “loss of faith” scenario, which in my experience seems atypical. I’m happy to type it up if anyone’s interested, but so far the lack of action on comments I make to old posts has me thinking that could be a spectacularly wasted effort. Vote, comment, or pm to show interest.
Ahh, thanks.
Causal knowledge is required to ensure success, but not to stumble across it. Over time, noticing (or stumbling across if you prefer) relationships between the successes stumbled upon can quickly coalesce into a model of how to intervene. Isn't this essentially how we believe causal reasoning originated? In a sense, all DNA is information about how to intervene that, once stumbled across, persisted due to its efficacy.
All these conclusions seem to require simultaneity of causation. If earthquakes almost always caused recessions, but not until one year after the earthquake; and if recessions drastically increase the number of burglars, but not until one year after the recession; then drawing any of the conclusions you made from a survey taken at a single point in time would be entirely unwarranted. Doesn't that mean you’re essentially measuring entailment rather than causation via a series of physical events which take time to occur?
Also, the virtue theory of metabolism is so ridiculous that it seems only to be acting as a caricature here. Wouldn't the theory that “exercise normally metabolises fat and precursors of fat, reducing the amount of weight put on” result in a much more useful example? Or is there a subtext I'm missing here, like the excessive amount of fat-shaming done in many of the more developed nations?
Great post! If this is the beginning of trend to make Less Wrong posts more accessible to a general audience, then I'm definitely a fan. There's a lot of people I'd love to share posts with who give up when they see a wall of text.
There are two key things here I think can be improved. I think they were probably skipped over for mostly narrative purposes and can be fixed with brief mentions or slight rephrasings:
You won't get a direct collision between belief and reality - or between someone else's beliefs and reality - by sitting in your living-room with your eyes closed.
In addition to comparison to external data such as experimental results, there are also critical insights on reality to be gained by armchair examination. For example, armchair examination of our own or others’ beliefs may lead us to realise that they are self-contradictory, and therefore that it is impossible for them to be true. No experimental results needed! This is extraordinarily common in mathematics, and also of great personal value in everyday thinking, since many cognitive mistakes lead directly to some form of internal contradiction.
And yet it seems to me - and I hope to you as well - that the statement "The photon suddenly blinks out of existence as soon as we can't see it, violating Conservation of Energy and behaving unlike all photons we can actually see" is false, while the statement "The photon continues to exist, heading off to nowhere" is true.
It's better to say that the first statement is unsupported by the evidence and purely speculative. Here's one way that it could in fact be true: if our world is a simulation which destroys data points which won’t in any way impact the future observations of intelligent beings/systems. In fact, that’s an excellent optimisation over an entire class of possible simulations of universes. There would be no way for us to know this of course (the question is inherently undecideable) but it could still happen to be true. In fact, we can construct extremely simply toy universes for which this is true. Undecideability in general is a key consideration that seems missing from many Less Wrong articles, especially considering how frequently it pops up within any complex system.
Good quote, but what about the reality that I believe something? ;) The fact that beliefs themselves are real things complicates this slightly.
Also known as the "people can't remember things without distinctive features" phenomenon. Still interesting to note their behaviours in the situation though.
I understand the point Eliezer's trying to make here. However, you (whoever's reading this) could not convince me that ss0 + ss0 =sss0 in Peano arithmetic (I define the scenario in which my mind is directly manipulated so that I happen to believe this not to constitute "convincing me"). Here's why I believe this position to be rational:
A)In order for me to make this argument, I have to presume communication of it. It's not that I believe the probability of that communication to be 1. Certainly many people might read this comment and not know Peano arithmetic, misunderstand my language, not finish reading, etc. etc. etc. and the probability of this is nontrivial. However, arguments are directed at the possible worlds in which they are understood.
B)Communication of "ss0 + ss0 =" as a statement of Peano arithmetic already fully constrains the answer to be "ssss0" simply by virtue of what these symbols mean. That is to say, that having understood these symbols and Peano arithmetic, no further experience is necessary to know that "sss0" is wrong. Mental flaws at any point in this process or understanding are possible, but they exist only within possible worlds in which communication of these ideas does not actually occur because to think that "ss0 + ss0 = sss0" is to misunderstand Peano arithmetic, and understanding Peano arithmetic is a prerequisite for understanding a claim about it.
Therefore
C)There is no possible world within which I can be convinced of the properly communicated concept "ss0 + ss0 = sss0". Of course, this doesn't mean there's no possible world in which I can be convinced that I am experiencing a neurological fault or being manipulated, or that there are no possible worlds in which I happen to wrongly believe that ss0 + ss0 = sss0. It's just that someone experiencing a neurological fault or being manipulated is not the same thing as someone being convinced.
A similar argument holds for the impossibility of me convincing myself that ss0 + ss0 = sss0. I understand ss0 + ss0 = ssss0 in Peano arithmetic well enough that I can review in a very short period of time why it must be so. Thus you would literally have to make me forget that I know this in order to have me believe otherwise, which hardly counts as "convincing." This does not mean that I am presuming mental errors or Dark Lords of the Matrix to be impossible. For clarification, here's what a runthrough of me experiencing what Eliezer proposes would look like:
I get up one morning, take out two earplugs, and set them down next to two other earplugs on my nighttable, and noticed that there were now three earplugs, without any earplugs having appeared or disappeared—in contrast to my stored memory that 2 + 2 was supposed to equal 4.
Because that stored memory entails an understanding of why, I run through those reasons. If they're incomplete this constitutes me "forgetting that I know this." (It does not mean that I don't know this now. Right now I do.) Therefore I don't have a "stored memory that 2 + 2 was supposed to equal 4." I have an incomplete stored memory which tries to say something about 2, +, =, and 4 (if my personality were intact I would probably try and re-derive the missing parts of it, after calling 911). Either way I identify a cognitive fault. In real life waking up to this my most likely suspect would be that my experience of one earplug disappearing was deleted before I processed it, but there are lots of other possibilities as well. If I repeated the experiment multiple times I would consider either a systematic fault or "being messed with" at a fundamental level.
When I visualize the process in my own mind, it seems that making XX and XX come out to XXXX requires an extra X to appear from nowhere
Still presuming an intact line of reasoning saying why this must not be so, I would again identify a cognitive fault, and a pretty cool one at that. Something this intricate might well leave me suspecting Dark Lords of the Matrix as a nontrivial possibility, provided all other cognitive functions seemed fully intact. Still wouldn't be as likely as a weird brain fault, though. I would definitely have fun investigating this.
I check a pocket calculator, Google, and my copy of 1984 where Winston writes that "Freedom is the freedom to say two plus two equals three."
Dark Lords of the Matrix bump higher, but Psychosis has definitely leapt into the front of the pack.
I could keep going, of course. These last few presume I can still reason out something like Peano arithmetic. If I can't incidentally, then of course they look different but I still don't think it would be accurate to describe any possible outcome as "me being convinced that 2 + 2 = 3." If you run all the way down the list until you literally delete all things that I know and all ways I might obtain them, I would describe that as a possible universe in which "me" has been deleted. The strict lower bound on where I can still stumble across my cognitive fault and/or manipulation is when my reasoning ability is no longer Turing complete. This essentially requires the elimination of all complex thought, though of course making it merely unlikely for me to stumble upon the fault is much easier--just delete everything I know about formal mathematics, for example.
tl;dr
I agree with most of what Eliezer is saying, but wouldn't say that I could be convinced 2 + 2 = 3. Does this make my belief unconditional? Dependant on me understanding what 2 + 2 = 3 means, maybe it does. Maybe an understanding of 2, +, =, 3, and 4 necessitates 2 + 2 = 4 for a rational mind, and any deviation from this, even in internal mental processes, would be identifiable as a fault. After all, you can run a software program to detect flaws in some computer processors.