Yet more "stupid" questions

I used to have Age of Empires dreams. I've even had Tetris dreams.

I get something similar, in that I frequently lose my perspective as a humanoid actor in my dreams. It appears that in my own dreams I am more or less incapable of simulating another living being without subjectively experiencing that beings thoughts and emotions at the same time. Perhaps for that reason my dreams are usually of empty wilderness or stars flying around. However, a few times per month I wake up very confused at just being one person because while asleep I experienced the thoughts of multiple individuals simultaneously, including whatever emotions they felt and their relative lack of information of each others perspectives. The maximum number of people I've been at once was seven, where three beings were fighting another two beings to save the other three from being tortured.

I've seen top-down perspectives in dreams, such as those in 2D RPGs. I feel like I'm playing a video game, but I don't have an awareness of a controller or anything; the characters just do what I tell them, and the "screen" is my entire visual field. (The actual experience of playing a video game tends to be similar: I almost never think about the controller or my hand; I just make stuff happen.) I also tend not to have much of a kinesthetic sense in dreams I remember, either.

Another weird thing: Everything I try to type in dreams is invariably misspelled. Once, in a dream, I was trying to Google something, but the text I was "typing" in the search bar kept changing pretty much at random. Only the letters that I'm "looking at" during any given moment stay what they are.

Yes to both.

It's very common for me to be human but different (child, woman, or very different looking man). Most common non-human forms are non-human-ape, wolf, or bird. Sometimes it's an imaginary monster of some sort. But dreaming in non-human forms is generally fairly rare.

Most common non-embodied perspectives are either eye-level-view, television view-style, or looking down from a distance. In these cases, I'll either self-identify with one of the bodies, or simply be an observer. This frequently switches mid-storyline.

I've had similar dreams.

In general I don't think I'm aware of my self/body in dreams. Occasionally I'm different people but dont notice.

I've had some dreams like that - a few dreams were specifically of the game Supreme Commander, and I also occasionally am in third-person in dreams as if I was watching from the screen of a third-person game - I don't think it's really "very different from real life", it's close to the experience of being immersed in a videogame, it's just that rather than overlooking details you're not paying attention to, those details simply don't exist because it's a dream.

One data point here. I've taken a few low-to-lowish dose trips. I'm still the same skeptic/pragmatist I was.

When I'd see the walls billowing and more detail generating out of visual details, I didn't think "The universe is alive!" I thought "my visual system is alive".

I did have an experience which-- to the extent I could put it into words-- was that my sense of reality was something being generated. However, it didn't go very deep-- it didn't have aftereffects that I can see. I'm not convinced it was false, and it might be worth exploring to see what's going on with my sense of reality.

I won't be able to do it justice in words, but I like to try.

If you value your current makeup as a "rationalist" - LSD will not necessarily help with that. Whatever your current worldview, it is not "the truth", it is constructed, and it will not be the same after you come down.

You can't expect a trip to do anything in particular, except maybe blow your mind. A trip is like finding out you were adopted. It's discovering a secret hidden in plain sight. It's waking up to realize you've never been awake before - you were only dreaming you were awake. It's finding out that everything familiar, everything you took for granted, was something else all along, and you had no idea.

No matter how much you've invested in the identity of "rationalist", no matter how much science you've read... Even if you know how many stars there are in the visible universe, and how many atoms. Even if you've cultivated a sense for numbers like that, real reality is so much bigger than whatever your perception of it is. I don't know how acid works, but it seems to open you in a way that lets more of everything in. More light. More information. Reality is not what you think it is. Reality is reality. Acid may not be able to show you reality, but it can viscerally drive home that difference. It can show you that you've been living in your mind all your life, and mistaking it for reality.

It will also change your sense of self. You may find that your self-concept is like a mirage. You may experience ego-loss, which is like becoming nobody and nothing in particular, only immediate sensory awareness and thought, unconnected to what you think of as you, the person.

I don't know about health dangers. I never experienced any. Tripping does permanently change the way you view the world. It's a special case of seeing something you can't un-see. Whether it's a "benefit" ... depends a lot on what you want.

I don't imbibe (for that matter, pretty much anything stronger than caffeine), so I can't offer any information about the experience of its affects on rationality.

From the literature, it has a relatively high ratio of activity threshold to lethal dose (even assuming the lowest supported toxic doses), but that usually doesn't include behavior toxicity. Supervision is strongly recommended. There's some evidence that psychoactive drugs (even weakly psychoactive drugs like marijuana) can aggravate preexisting conditions or even trigger latent conditions like depression, schizophrenia, and schizoid personality disorder.

Another data point here. I've done LSD a couple of times, and didn't find the experience "spiritual" at all.

The experience was mostly visual: illusion of movement in static objects when eyes open, and intense visualization when eyes closed. It's hard to describe these images, but it felt like my visual cortex was overstimulated and randomly generated geometric patterns intertwined with visual memories and newly generated constructs and sceneries. This all happened while travelling through a fractal-like pattern, so I felt the word "trip" was quite fitting. The trip didn't seem to affect my thinking much during or after.

I can see why a susceptible (irrational) mind could find this chemical alteration of consciousness a godly revelation, but I can't imagine taking the stuff for anything else than entertainment purposes. A couple of friends of mine had similar experiences.

LSD is known to cause persistent psychosis, apparently in people who already have latent or diagnosed mental health problems. This is what they teach in my med school, but the epidemiology of the phenomenom was left vague.

Datapoint: another halluciogen, ketamine, has been shown to effectively treat depression. Not sure if mechanisms of LSD are similar.

Based on the descriptions of thoughtful, educated people who practise hypnosis, it seems useful to think of it as a "suite" of psychological effects such as suggestion, group conformity, hype, etc., rather than a single coherent phenomenon.

Not sure exactly what you want to know here, but here are a few basic points:

1. Hypnotized people are not unconscious, rather they are fully awake and focused.

2. Brain scans don't show any signs of abnormal brain activity during hypnosis.

3. Some psychologists argue hypnotized people are just fulfilling the socially expected role for a hypnotized person.

Do you know how normal hypnotism works?

No it does not. Aspirin reduces the risk of heart attacks and strokes but also causes adverse outcomes - most importantly by raising the risk of gastro-intestinal bleeds. For the typical person in their mid twenties the risk of a heart attack or stroke is so low that the benefit of aspirin will be almost nil, the absolute value of intervening will be vanishingly small even though the proportional decrease in risk stays the same.

There are many possible effects of taking low dose aspirin other than those described so far - it may reduce the risk of colon cancer, for instance, but there are so many possible adverse outcomes too. Cyclooxygenase - the enzyme targeted by aspirin - is involved in many housekeeping functions throughout the body in particular the kidneys, stomach and possibly erectile tissue.

Studies examining risk versus benefit for low dose aspirin treatment have found that a cardiovascular risk of about 1.5%/year is necessary for the benefits of aspirin to outweigh the ill effects. Whilst no studies have been conducted on healthy young individuals I don't think such studies should be conducted, given that studies in those at a much higher cardiovascular risk than someone in their twenties have returned disappointing results we should not expect any great benefit from such a treatment. Indeed young people, men in particular, are much more likely to experience trauma than a cardiovascular event and patients taking low dose aspirin are much more likely to experience severe bleeding after trauma.

See this article for more information: http://www.sciencebasedmedicine.org/aspirin-risks-and-benefits/

Cultivate a sense of warm satisfaction every time you avoid a pointless online debate.

Really think about how very much is wrong on the internet compared to your capacity to try to correct it. I think this might be a case of cultivating scope sensitivity.

Or (which is what I think I do) combine that with a sense that giving a little shove towards correctness is a public service, but it isn't a strong obligation. This tones the compulsion down to a very moderate hobby.

For me debating with people on LessWrong somehow cured the syndrome. Now when I see a political debate among non-LessWrongians, the participants seem like retarded people -- I no longer expect them to be reasonable; I don't even expect them to be able to understand logical arguments and process them correctly.; I don't feel any hope of conveying anything meaningful to any of them. (At best we could have an illusion of understanding.) Speaking with them would be like speaking with a rock; certainly not tempting.

I am not saying this is a correct model of the world. It is probably exaggerated a bit. Just explaining that this is how I feel, and this is what cured the syndrom.

These days the syndrom manifests mostly when speaking with someone whom I consider they could be rational. If they feel like a potential LW candidate. It usually ends with me revising my opinion about the candidate, and silently stopping.

So, for me the cure is feeling that the inferential distance between typical internet discussion and rational discussion is so huge that I don't have a chance to overcome it in one debate.

Realize that it's not their fault, they are just automatons with faulty programming.

I just became unwilling to devote the effort to replying.

So are people conflating utility maximization and utilitarianism

Often, yes.

It's true that people often conflate utilitarianism with consequentialism, but I don't think that's what's going on here. I think it is quite reasonable to include under utilitarianism moral theories that are pretty close, like weighting people when aggregating. If people think that raw utilitarianism doesn't describe human morality, isn't it more useful for the term to describe people departing from the outpost, rather than the single theory? Abstract values that are not per-person are more problematic to include in the umbrella, but searching for "free" in that post doesn't turn up an example. If your definition is so narrow that you reject Nozick's utility monster as having to do with utilitarianism, then your definition is too narrow. Also, the lack of a normalization means that giving everyone "the same weight" does not clearly pin it down.

This confused me for a long time too. I ultimately came to the conclusion that "utilitarianism" as that word is usually used by LessWrongers doesn't have the standard meaning of "an ethical theory that holds some kind of maximization of utils in the world to be the good", and instead uses it as something largely synonymous with "consequentialism".

Update: I decided on Permutation City, and was unable to put it down until it was done. I am very happy with the book. I am a lot more convinced now that I will eventually read almost all of these, so the order doesn't matter as much.

Terry Pratchett's Discworld series. I recommend starting with Mort (the fourth book published). The first two books are straight-up parodies of fantasy cliches that are significantly different from what comes afterward, and the third book, Equal Rites, I didn't care for very much. Pratchett said that Mort was when he discovered plot, and it's the book that I recommend to everyone.

Well, if you liked MoR, there are the two other Big Rationalist Fanfics:

• http://www.fimfiction.net/story/62074/friendship-is-optimal (don't judge it because it's about ponies - it's really good even if you dont like them) (if yo like it, don't forget to check out the meta-fanfics)
• http://luminous.elcenia.com/story.shtml (as good as the thing it's parodying is terrible)

Also in a similar style: http://www.sagaofsoul.com/

Then there's the scifi classics, if you're willing to shell out some money. no links for these. Here are a few good ones to get you started

• Permutation City
• Accelerando
• Diaspora
• Fire Upon the Deep

This should be enough to get you started. I can give you MUCH more if you want to and maybe tell me some other things you like. Finding stuff like this to specification is basically what I do.

"smart characters that win"

Miles Vorkosigan saga, Ender's Game, anything by Neal Stephenson.

I am a mathematician and a Bayesianist with an emphasis on the more theoretical side of rationality. I like smart characters that win. I looked at some recommendations on other topics, but there are too many options.

Give Neal Stephenson a go. Snow Crash and Cryptonomicon are good starting points.

First is probably Bujold, specifically her Miles Vorkosigan series.

I think of Vinge more in terms of awesome author than awesome characters, but he does have some pretty impressive characters.

Lee Child has an intelligent good guy and intelligent associates vs. intelligent bad guys. (Not sf.)

The First Law trilogy by Joe Abercrombie. No promises on the characters, most of them are not so rational, but you'll see why I said it by the end. There are more books in the same setting with some of the same characters if you like them. The first book is mostly setup but it is great after that.

Re "smart characters that win," I recommend these from my random reading history:

The Pillars of the Earth and A World without Endby Ken Follett

River God by Wilbur Smith

Singularity Sky and Iron Sunrise by Charles Stross

And Then There Were None by Agatha Christie

SF; Go to the amazon kindle store, read the first chapters (free samples) of:

Vernor Vinge, a fire on the deep. Finest example of classical IE; space ships, politics and aliens! SF there is.

Lois Mcmaster Bujold: A large sample of the first book in the Vor saga. http://www.baen.com/chapters/W200307/0743436164.htm?blurb

If you like Harry for being a high-competence chaos magnet, this should scratch that itch in just the right spot.

It's not specifically rationalist, but Dune is what first comes to mind for "smart characters that win", at least in the first book.

I recommend pretty much anything by Jack Vance. If you like fantasy settings, read "Lyonesse", "Cugel's Saga" and "Rhialto the Marvellous". If you like sci-fi settings, try "Araminta Station" , "Night Lamp" and "Alastor". For a quaint mix of the two, try "Emphyrio" or "Languages of Pao". Vance wrote a bunch of great stuff, so if you like his first book, you have heaps more to look forward to.

Also "Name of the Wind" and "Wise Man's Fear" by Patrick Rothfuss are pretty good.

I also second "Ender's Game".

The "show help" box at the lower right of the comment field gives you information on the markdown methods for emphasis and links and such.

I'm pretty sure that you just use links to your other posts for follow up links, unless I'm missing something about your question.

MWI or non-ontological collapse gets you to a place where you can even entertain the notion that the framework of Quantum Mechanics is correct and complete, so that:

• you can stop worrying about bogus unphysical theories that you'd only invent in an attempt to make things look normal again, and
• you're more comfortable with working with larger superpositions.

Because under most current legal system this is called "murder".

Assuming that you become some kind of superintelligence, I'd expect you to find better ways of amusing yourself, yes; especially if you're willing and able to self-modify.

Conservation of energy can be derived from Lagrangian mechanics from the assumption that the Lagrangian is constant over time. That is equivalent to saying that the dynamics of the system do not change over time. If the mechanics are changing over time, it would certainly be more difficult to predict future states, and one could imagine the mechanics changing unpredictably over time, in which case future states could be unpredictable as well. But now we don't just have physics that changes in time, we have physics that changes randomly.

I think I find that thought more troubling than the lack of free will.

(I know of no reason why any further conservation laws would break in a universe such as that, so long as you maintain symmetry under translations, rotations, CPT, etc. Time-dependent Lagrangians are not exotic. For example, a physicist might construct a Lagrangian of a system and include a time-changing component that is determined by something outside of the system, like say a harmonic oscillator being driven by an external power source.)

I don't see any direct link between determinism and conservation of energy. You can have one or the other or both or none. You could have laws of physics like "when two protons collide, they become three protons", determinist but without conservation of energy.

As for "libertarian free will" I'm not sure what you mean by that, but free will is concept that must be dissolved, not answered "it exists" or "it doesn't exist", and anyway I don't see the link between that and the rest.

Liouville's theorem is more general than conservation of energy, I think, or at least it can hold even if conservation of energy fails. You can have a system with a time-dependent Hamiltonian, for instance, and thus no energy conservation, but with phase space volume still preserved by the dynamics. So this would be a deterministic system (one where phase space trajectories don't merge) without energy conservation.

As for the minimum necessary set of conservation laws that must be knocked out to guarantee non-determinism, I'm not sure. I can't think of any a priori reason to suppose that determinism would crucially rely on any particular set of conservation laws, although this might be true if certain further constraints on the form of the law are specified.

Objective morality in one (admiitedly rather long) sentence: For any moral dilemma, there is some particular decision you would make after a thousand years of collecting information, thinking, upgrading your intelligence, and reaching reflective equilibrium with all other possible moral dilemmas; this decision is the same for all humans, and is what we refer to when we say that an action is 'correct'.

At least some of the prominent regulars seem to believe in objective morality outside of any FAI context, I think (Alicorn? palladias?).

The connotations of "objective" (also discussed in the other replies in this thread) don't seem relevant to the question about the meaning of "correct" morality. Suppose we are considering a process of producing an idealized preference that gives different results for different people, and also nondeterministically gives one of many possible results for each person. Even in this case, the question of expected ranking of consequences of alternative actions according to this idealization process applied to someone can be asked.

Should this complicated question be asked? If the idealization process is such that you expect it to produce a better ranking of outcomes than you can when given only a little time, then it's better to base actions on what the idealization process could tell you than on your own guess (e.g. desires). To the extent your own guess deviates from your expectation of the idealization process, basing your actions on your guess (desires) is an incorrect decision.

A standard example of an idealization dynamic is what you would yourself decide given much more time and resources. If you anticipate that the results of this dynamic can nondeterministically produce widely contradictory answers, this too will be taken into account by the dynamic itself, as the abstract you-with-more-time starts to contemplate the question. The resulting meta-question of whether taking the diverging future decisions into account produces worse decisions can be attacked in the same manner, etc. If done right, such process can reliably give a better result than you-with-little-time can, because any problem with it that you could anticipate will be taken into account.

A hypothetical idealization dynamic may not be helpful in actually making decisions, but its theoretical role is that it provides a possible specification of the "territory" that moral reasoning should explore, a criterion of correctness. It is a hard-to-use criterion of correctness, you might need to build a FAI to actually access it, but at least it's meaningful, and it illustrates the way in which many ways of thinking about morality are confused.

(As an analogy, we might posit the problem of drawing an accurate map of the surface of Pluto. My argument amounts to pointing out that Pluto can be actually located in the world, even if we don't have much information about the details of its surface, and won't be able to access it without building spacecraft. Given that there is actual territory to the question of the surface of Pluto, many intuition-backed assertions about it can already be said to be incorrect (as antiprediction against something unfounded), even if there is no concrete knowledge about what the correct assertions are. "Subjectivity" may be translated as different people caring about surfaces of different celestial bodies, but all of them can be incorrect in their respective detailed/confident claims, because none of them have actually observed the imagery from spacecraft.)

I came to the metaethics sequence an ethical subjectivist and walked away an ethical naturalist. I've mostly stopped using the words "objective" and "subjective", because I've talked with subjectivists with whom I have few to no substantive disagreements. But I think you and I do have a disagreement! How exciting.

I accept that there's something like an ordering over universe configurations which is "ideal" in a sense I will expand on later, and that human desirability judgements are evidence about the structure of that ordering, and that arguments between humans (especially about the desirability of of outcomes or the praiseworthiness of actions) are often an investigation into the structure of that ordering, much as an epistemic argument between agents (especially about true states of physical systems or the truth value of mathematical propositions) investigates the structure of a common reality which influences the agents' beliefs.

A certain ordering over universe configurations also influences human preferences. It is not a causal influence, but a logical one. The connection between human minds and morality, the ideal ordering over universe configurations, is in the design of our brains. Our brains instantiate algorithms, especially emotional responses, that are logically correlated with the computation that compresses the ideal ordering over universe configurations.

Actually, our brains are logically correlated with the computations that compress multiple different orderings over universe configurations, which is part of the reason we have moral disagreements. We're not sure which valuation - which configuration-ordering that determines how our consequential behaviors change in response to different evidence - which valuation is our logical antecedent and which are merely correlates. This is also why constructed agents similar to humans, like the ones in Three Worlds Collide, could seem to have moral disagreements with humans. They, as roughly consequentialist agents, would also be logically influenced by an ordering over universe configurations, and because of similar evolutionary pressures might also developed emotion-type algorithms. The computations compressing the different orderings, morality versus "coherentized alien endorsement relation" would be logically correlated, would be partially conditionally compressed by knowing the value of simpler computations that were common between the two. Through these commonalities the two species could have moral disagreements. But there would be other aspects of the computations that compress their orderings, logical factors that would influence one species but not the other. These would naively appear as moral disagreements, but would simply be mistaken communication: exchanging evidence about different referents while thinking they were the same.

But there are other sources of valuation-disagreement than being separate optimization processes. Some sources of moral disagreement between humans: We have only partial information about morality, just as we can be partially ignorant about the state of reality. For example, we might be unsure what long term effects to society would accompany the adoption of some practice like industrial manufacturing. Or even if someone in the pre-industrial era had perfect foresight, they might be unsure of how their expressed preferences toward that society would change with more exposure it. There are raw computational difficulties (unrelated to prediction of consequences) in figuring out which ordering best fits our morality-evidence, since the space of orderings over universe configurations is large. There are still more complicated issues with model selection because human preferences aren't fully self endorsing.

Anyway, I've been using the word "ideal" a lot as though multiple people share a single ideal, and it's past time I explained why. Humans share a ton of neural machinery and have a spatially concentrated origins, both of which mean closer logical-causal influences to their roughly-consequential reasoning. We have so much in common that saying "Pah, nothing is right. It's all just subjective preferences and we're very different people and what's right for you is different from what's right for me" seems to me like irresponsible ignorance. We've got like friggin' hundreds of identical functional regions in our brains. We can exploit that for fun and profit. We can use interpersonal communication and argumentation and living together and probably other things to figure out morality. I see no reason to be dismissive of others' values that we don't sympathize with simply because there's no shiny morality-object that "objectively exists" and has a wire leading into all our brains or whatever. Blur those tiniest of differences and it's a common ideal. And that commonality is important enough that "moral realism" is a badge worth carrying on my identity.

People are often wrong about what their preferences are + most humans have roughly similar moral hardware. Not identical, but close enough to behave as if we all share a common moral instinct.

When you make someone an argument and they change their mind on a moral issue, you haven't changed their underlying preferences...you've simply given them insight as to what their true preferences are.

For example, if a neurotypical human said that belief in God was the reason they don't go around looting and stealing, they'd be wrong about themselves as a matter of simple fact.

-as per the definition of preference that I think makes the most sense.

-Alternatively, you might actually be re-programming their preferences...I think it's fair to say that at least some preferences commonly called "moral" are largely culturally programmed.

I just assumed it meant "My extrapolated volition" and also "your extrapolated volition" and also the implication those are identical.

I wrote a post to try to answer this question. I talk about "should" in the post, but it applies to "correct" as well.

Here is a decent discussion of objective morality.

What's up with that?

The usual Typical Mind Fallacy which is really REALLY pervasive.

What are the specs of muscles?

There are books and papers on the physiology of exercise, in particular on how muscles use energy in different regimes. For a basic intro check Crossfit, for more details you can look at e.g. Body By Science.

trying to optimize

What are you trying to optimize for?

Similarly: What are a qubit's specs? I would like to be able to think about what class of problem would be trivial with a quantum computer.

I read fiction about 1/3 of the time and nonfiction 2/3s of the time. When reading non-fiction I often spend idle moments in my day lost in abstract thought about concepts related to the non-fiction book I'm reading. I've noticed when reading novels, I'm far more observant about people in my personal life and their thoughts and motivations. This is especially true when reading works with thoughtful and observant POV characters (especially detective fiction and mystery novels). I think fiction, like music, can serve to frame your mind-state in a certain way.

• It has been useful for manipulating my mood and general mindset. ("My life would be more amusing right now if I felt a bit nervous and unsure of reality. Better go read something by Philip K. Dick." Or "I would be more productive if I were feeling inspired by the grandness of the scientific endeavor. Better go read some Golden Age SF.")
• It is useful for understanding what certain situations feel like without going through them yourself, and therefore can help you empathize with people in those situations whose behavior otherwise does not make sense to you. Memoirs and other nonfiction can also do this, but it's easier to find well-written fiction than well-written nonfiction, and for this purpose the writing must be very good.

Notion that fiction increases empathy has been making the rounds. Not an area I've researched heavily, I am intrigued but skeptical.

most of my comments take less than a minute to write.

Depends on the comment. Mine are almost all rewrites, so anything that's not a short answer to a simple factual question takes me at least a couple of minutes. The upper bound is probably ≈2 hours.

If I remember rightly, this one took almost that long, and would've taken longer if I'd tried to polish it and not have it end in disjointed bullet points. There are quite a few reasons why that comment was so time-consuming: it was lengthy; it was a response to criticism, so I wanted to make it very obviously correct; I wanted to refer to lots of examples and sources, which means deciding which bits of which sources to quote, and hitting up Google & Wikipedia; I wanted to cover quite a lot of ground so I had to spend more time than usual squeezing out verbiage; and I had to stop & think periodically to check everything I was saying came together coherently.

Sometimes I write a few paragraphs, realize I've written myself into a corner, then decide to tear everything down and start over from a different angle. (Or I decide it's not worth the effort and refrain from saying anything.) That happened with this comment, so it wound up taking something like an hour.

This comment, by contrast, has only needed about half an hour to write because it's mostly based on introspection, isn't that long, isn't communicating anything complex, won't be controversial, isn't optimized for transparency, and turns out not to have needed any full-scale rewrites.

I also think I'm a slow writer by LW standards. (Unsurprisingly?)

Because doing so will lead to worse outcomes on average. Over a long series of events, someone who just follows the math will do better than someone who is risk-averse wrt to 'utility'. Of course, often our utility functions are risk-averse wrt to real-world things, because of non-linear valuation - e.g, your first $100,000 is more valuable than your second, and your first million is not 10x as valuable as your first $100,000.

It does seem like an interesting question. But the most obvious flaw is that we still don't have the starting point - software does what we tell it to do, not what we want, which is usually different - and I don't immediately see any way to get there without super-intelligence.

Holden Karnofsky proposed starting with an Oracle AI that tells us what it would do if we gave it different goal systems. But if we avoided giving it any utility function of its own, the programmers would need to not only think of every question (regarding every aspect of "what it would do"), but also create an interface for each sufficiently new answer. I'll go out on a limb and say this will never happen (much less happen correctly) if someone in the world can just create an 'Agent AI'.

How do we make an AI that obediently performs as we want it to, but does so smarter, while maintaining it's obedience?

Depends on what you mean by "smarter"? It is merely good at finding more efficient ways to fulfill your wish... or is it also able to realize that some literal intepretations of your wish are not what you actually want to happen (but perhaps you aren't smart enough to realize it)? In the latter case, will it efficiently follow the literal intepretation?

Some systems exhibit a butterfly effect (a.k.a. chaos); some don't. The butterfly effect is where (arbitrarily) small changes to the conditions of the system can totally change it's future course. The weather is a good example of this. The change caused by a butterfly flapping it's wing differently will amplify itself until the entire Earth's weather is different from what it would have been. But other systems aren't like that. They're more "stable". For example if you change the position of any individual atom in my computer it won't make any difference to the computations I'm running. Other things are predictable just because we don't give time for any changes to develop. For example ball collisions are predictable, but if we study many ball collisions in a row, like a billiards "trick shot", then hitting the initial ball slightly differently will make a huge difference.

You ask about quantum events. For chaotic systems, deviations caused by quantum events will indeed cause a butterfly effect.

So whether or not the brain is predictable depends on to what extent it's chaotic, and to what extent it's stable. I suspect that it's chaotic, in the sense that a small tweak to it could totally change the way a thought process goes. But over time my brain will be predictable "on average". I'll behave in ways matching my personality. Similarly a butterfly flapping it's wings might change when it rains, but it'll still rain more in Bergen than the Sahara.

I don't think this says much about Omega problems. Quantum butterfly effects will (I suspect) stop Omega exactly simulating my thought process, but I reckon it could still predict my choice with very high confidence just by considering my most likely lines of thought.

The butterfly effect kicks in wherever there's something unstable - whenever there's a system where little changes grow. Billiards balls do this, for instance, which is why it's harder to hit the cue so it hits the 4 so it hits the 1 so it hits the 5 than to hit the cue so it hits the 5 (assuming the same total ball travel distance).

Quantum noise is no less capable of doing this than anything else. The reason macro objects look solid has little to do with special cancellation and a lot to do with how tightly bound solid objects are. I suppose that's a special case of cancellation, but it's a really special case.

Omega-like problems are hypotheticals, and speaking of quantum indeterminacy in respect to them is fighting the hypothetical. Some versions word it so if Omega can't get a reliable answer he doesn't even play the game, or withholds the money, or kicks you in the shins or something - but those are just ways of getting people to stop fighting the hypothetical.

I don't believe I am explaining MWI instead of arguing against it... whatever has this site done to me? Anyway, grossly simplified, you can think of the matter as being conserved because the "total" mass is the sum of masses in all worlds weighted by the probability of each world. So, if you had, say, 1kg of matter before a "50/50 split", you still have 1kg = 0.5*1kg+0.5*1kg after. But, since each of the two of you after the split has no access to the other world, this 50% prior probability is 100% posterior probability.

Also note that there is no universal law of conservation of matter (or even energy) to begin with, not even in a single universe. It's just an approximation given certain assumptions, like time-independence of the laws describing the system of interest.

No. It's not that kind of many-ness.

For a huge oversimplification:

The cosmos is a big list of world-states, of the form "electrons in positions [(12.3, -2.8, 1.0), (0.5, 7.9, 6.1), ...] and speeds [...] protons in positions...". To each state, a quantum amplitude is assigned.

The laws of physics describes how the quantum amplitude shifts between world states as time goes by (based on speed of particles and various basic interactions ....).

Conservation of matter says that for each world state, you can compute the amount of matter (and energy) inside, and it stays the same.

No, at least not in a technical mathematical-physics sense. "Conservation of matter", in mathematical physics, translates to the Hamiltonian operator being conserved, and that happens in quantum physics and a fortiori in all its plausible philosophical interpretations. In concrete, operationalist terms, this implies that an observer measuring the energy of the system at different times (without disturbing it in other way in the meantime) will see the same energy. It doesn't imply anything about adding results of observations in different MWI branches (which is probably meaningless).

For example if you have an electron with a given energy and another variable that "branches", then observers in each branch will see it with the same energy it had originally, and this is all the formal mathematical meaning of "conservation" requires. The intuition that the two branches together have "more energy" that there was initially and this is a conservation problem is mixing pictorial images used to describe the process in words, with the technical meaning of terms.

I can tell you the details, but they don't really matter.

MWI has not been experimentally disproven. It all adds up to normality. Whatever observations we've made involving energy conservation are predicted by MWI.

Depends how you interpret it. If you say a new universe is created with every quantum decision, then you could argue that(though I've always treated conservation laws as being descriptive, not proscriptive - there's no operation which changes the net amount of mass-energy, so it's conserved, but that's not a philosophical requirement). But the treatment of many-worlds I see more commonly is that there's already an infinite number of worlds, and it's merely a newly-distinct world that is created with a quantum decision.

The deeper (and truer) version of "conservation of matter" is conservation of energy. And energy is conserved in many worlds. In fact, that's one of the advantages of many worlds over objective collapse interpretations, because collapse doesn't conserve energy. You can think of it this way: in order for the math for energy conservation to work out, we need those extra worlds. If you remove them, the math doesn't work out.

Slightly more technical explanation: The Schrodinger equation (which fully governs the evolution of the wavefunction in MWI) has a particular property, called unitarity. If you have a system whose evolution is unitary and also invariant under time translation, then you can prove that energy is conserved in that system. In collapse interpretations, the smooth Schrodinger evolution is intermittently interrupted by a collapse process, and that makes the evolution as a whole non-unitary, which means the proof of energy conservation no longer goes through (and you can in fact show that energy isn't conserved).

I say the extent to which he has "killed" people is dependent on how much he diverges the new universe.

As in, Person A has some value between "Dead" and "Alive" which depends on the extent to which they differ from Person A' as a result of the reset.

Oh! Is this your Hypothetical A?

Interesting! What happens to everyone else when the universe "resets"? Do they basically die?

looks like we've said a lot on that

When your desires contradict each other, so you can't satisfy all of them anyway.

For example, I want to eat as much chocolate as possible and move at little as possible, but I also want to have a long healthy life. Until the Friendly AI can satisfy all my desires via uploading or nanotechnology, I must sacrifice some of them for the sake of other ones.

Put a human with good social skills in a box, expose it to a representative sample of people of various sexualities and reward it when it guesses right; the human brains social functionality is a very powerful specialized Bayesian engine. :p

Alternatively, just take your own brain and expose it to a large representative sample of people of varying sexualities and only check what they were afterwards. Not quite as technically powerful, but more portable and you get some extra metadata.

Advice for a similar problem is here

In a universe made of atoms and the void, how could it be the one true objective morality to be gloomy and dress in black?

http://xkcd.com/167/

Why are you not a nihilist?

For the same reason why I don't just lie down and stop doing anything at all. Knowledge of the fact that there isn't any ultimate meaning doesn't change the fact that there exist things which I find enjoyable and valuable. The part of my brain that primarily finds things interesting and valuable isn't wired to make its decisions based on that kind of abstract knowledge.

Why are you even reading this comment? :-)

What is the refutation of nihilism, in a universe made of atoms and the void?

"Sure, there is no ultimate purpose, but so what? I don't need an ultimate purpose to find things enjoyable."

why have the philosophers not all been fired and philosophy abolished?

Philosophy is the study of interesting questions, and nihilism hasn't succeeded in making things uninteresting.

Before I can answer the question, I need to have some idea of what "nihilism" means in this context, because there are many different varieties of it. I assume this is the most common one, the one that proposes that life is meaningless and purposeless. If this isn't the kind of nihilism you're referring to, please correct me.

To answer the question, I'm not a nihilist because nihilism is conceptually mistaken.

For example, suppose there is a stick, a normal brown wooden stick of some length. Now, is that stick a meter long or not? Whether it is or isn't, that question is conceptually sound, because the concept of stick has the attribute "length", which we can compare to the length of a meter, Is the stick morally just? This question isn't conceptually sound, because "justice" isn't an attribute of a stick. A stick isn't just, unjust, or morally gray, it completely lacks the attribute of "justice".

How does this apply to life? If you ask whether life is meaningless, that presupposes that conceptually life can have a meaning in the same way a stick can be a meter long - that "meaning" is an attribute of life. However, meaning is informational - words have meanings, as do symbols and signals in general. When I say "apple", you can imagine an apple, or at least know what I'm talking about, which means that the word "apple" is meaningful to both of us. If I say "Colorless green ideas sleep furiously", it doesn't bring anything to mind, so that phrase is meaningless. Life lacks the attribute of "meaning", because it's not information that's being communicated. Therefore, to say "life has no meaning" is more similar to saying "the stick is unjust" than to "the stick is shorter than a meter".

That deals with "life is meaningless". How about "life is purposeless"? To answer that question, consider where purpose comes from - from using something to achieve a desire. For example, if I say "a hammer's purpose is to hammer in nails", what that really means is something more like "A hammer is well-suited for hammering in nails and is often used for that end". If I want to hammer in nails, then, for me, the purpose of a hammer becomes to hammer in nails. If I want to eat porridge with a hammer (something I don't recommend), then to me the purpose of a hammer becomes to move porridge from a plate to my mouth. You may assign the hammer either of those purposes, or an entirely different one. Each of us can even assign multiple purposes to the same object. The point is, purpose is not a property of an object on its own, but one that arises from it having a relation with a being that has some use for it.

So, when you ask "What, if any, is the purpose of life?" that question requires much clarification. The purpose of whose life, and to whom? Just as we can assign different purposes to a hammer, we can assign different purposes to a life. For example, the purpose of my life to me is to keep me around, as I wouldn't be able to experience things if I were dead. Other people may assign different purposes to my life. So, a life can be purposeless, but only if no one, including the possessor of the life, assigns any value to it (and that assignment of value is in a reflective equilibrium).

To summarize:

"Is life meaningless?" - "Wrong question, meaning isn't an attribute of life."

"Is life purposeless?" - "Purpose is subjective and assigned by beings with desires. It is impossible to make a blanket statement about life in general, but it is possible for a particular life to be purposeless, though it is unlikely. Most lives have at least one purpose assigned to them."

If you taboo the word "nihilism", the question almost answers itself.

Simple: You're allowed to have values even if they aren't hard-coded into the fabric of the universe.

This uncaring universe had a misfortune to evolve macroscopic structures who do care about it and each other, as a byproduct of their drive to procreate.

Define 'nihilism'.

Death - SMBC Theater

Listen to the last guy.

If there is none, why have the philosophers not all been fired and philosophy abolished?

Fired by whom?

For me, I am not a nihilist because nihilism is boring. Also nihilism is a choice about how to see things, choosing nihilism vs non-nihilism does not come from learning more about the world, it comes from choosing something.

I am at least a little bit of a nihilist, there is plenty that I deny. I deny god, and more importantly, I deny a rational basis for morality or any human value or preference. I behave morally, more than most, less than some, but I figure i do that because I am genetically programmed to do so, and there is not enough to be gained by going against that. So I feel good when I bring my dog to the park because he has been genetically programmed to hack in to the part of my brain that I use for raising my children when they are babies, and I get powerful good feelings when I succumb to the demands of that part of my brain.

It makes no more rational sense to embrace nihilism than to deny it. It is like picking chocolate vs. vanilla, or more to the point, like picking chocolate vs poop-flavored. Why pick the one that makes you miserable when it is no more or less true than the one that is fun?

What is the refutation of nihilism, in a universe made of atoms and the void?

Who told you the universe is made of atoms and the void?

A good quote on this:

It is true that the changes brought about by human action are but trifling when compared with the effects of the operation of the great cosmic forces. From the point of view of eternity and the infinite universe man is an infinitesimal speck. But for man human action and its vicissitudes are the real thing. Action is the essence of his nature and existence, his means of preserving his life and raising himself above the level of animals and plants. However perishable and evanescent all human efforts may be, for man and for human science they are of primary importance.

In other words, even though it's true that every war, every destroyed relationship, every wonderful interaction, and everything else that's ever occurred in history happened on the pale blue dot, most likely quite ephemeral in its existence by contrast to the rest of the universe, this doesn't change about the fact that we as humans are programmed to care about certain things--things that do exist at this time, however transient they would be from a universe perspective--and this is the source of all enjoyment and suffering. The goal is to be on the 'enjoyment' side of it, of course.

Nihilism is just a confusion, a failure to take seriously the maxim 'it all comes back to normalcy'.

I'm sorry if my kind ever confused you by saying things like "It is important that I make an impressive display in the lek", what I actually mean is "It is likely my intrinsic goals would be well met if I made an impressive display in the lek". There is an ommitted variable in the original phrasing. Its importance isn't just a function of our situation, it's a function of the situation and of me, and of my value system.

So I think the real difference between nihilists and non-nihilists as we may call them, is that non-nihilists [think they]have a clearer idea of what they want to do with their life. Life's purpose isn't written on the void, it's written within us. Nobody sane will argue otherwise.

Actually... "within".. now I think of it, the only resolute nihilist I've probed has terrible introspection relative to myself, and it took a very long time to determine this, introspective clarity doesn't manifest as you might expect. This might be a lead.

I am a machine bent on maxemizing the result of a function when run over the multiverse, that measures the amount of certain types of computation it is isomorphic to.

I'm a nicilist instead

I found myself experiencing a sort of "emotional nihilism" after de-converting from Christianity...

To your questions:

1. I don't know that I'm not, though I don't really define myself that way. I don't know if life or the universe has some ultimate/absolute/objective purpose (and I suspect it does not) or even what "purpose" really means... but I'm content enough with the novelty and intrigue of learning about everything at the moment that nihilism seems a bit bleak for a label to apply to myself. (Maybe on rainy days?)

2. I don't know. I'd also be interested to hear a good refutation. I suppose one could say "you are free to create your own meaning" or something like that...and then you'd have personally thwarted nihilism. Meh.

3. I gotta believe a good chunk of the world still believes in meaning of some kind, if for no other reason than their adherence to religion. This is an economic reason for the survival of philosophy and ongoing speculation about meaning -- Clergy are often are just philosophers with magical pre-suppositions & funny outfits.

And, practically speaking, it seems like purpose/meaning is a pretty good thing to stubbornly look for even when facing seemingly irrefutable odds.

Hm... maybe you could say the refutation of nihilism is the meaning you find in not giving up the search for meaning even though things seem meaningless?

I know they love meta concepts around here...

There's only two options here. Either the universe is made of atoms and void and a non-material Cartesian subject who experiences the appearance of something else or the universe is filled with trees, cars, stars, colours, meaningful expressions and signs, shapes, spatial arrangements, morally good and bad people and actions, smiles, pained expressions, etc, all of which, under the appropriate conditions, are directly perceived without mediation. Naturalism and skeptical reductionism are wholly incompatible: if it was just atoms and void there would be nothing to be fooled into thinking otherwise.

I'm not sure what you're talking about, but it might be rot13, a simple substitution system for avoiding spoilers.

Rot13.com will code and decode passages to and from rot13.

Local thinking about FAI is predicated on the assumption that an AI is probably capable of (and will initiate) extremely rapid self-improvement (the local jargon is "FOOMing," which doesn't stand for anything as far as I know, it just sounds evocative), such that it rapidly becomes a significantly superhuman intelligence, and thereafter all such decisions can profitably left up to the FAI itself.

Relatedly, local thinking about why FAI is important is largely predicated on the same assumption... if AIs will probably FOOM, then UFAI will probably irrecoverably destroy value on an unimaginable scale unless pre-empted by FAI, because intelligence differentials are powerful. If AIs don't FOOM, this is not so much true... after all, the world today is filled with human-level Unfriendly intelligences, and we seem to manage; Unfriendly AI is only an existential threat if it's significantly more intelligent than we are. (Well, assuming that things dumber than we are aren't existential threats, which I'm not sure is justified, but never mind that for now.)

Of course, if we instead posit either that we are incapable of producing a human-level artificial intelligence (and therefore that any intelligence we produce, being not as smart as we are, is also incapable of it (which of course depends on an implausibly linear view of intelligence, but never mind that for now)), or that diminishing returns set in quickly enough that the most we get is human-level or slightly but not significantly superhuman AIs, then it makes sense to ask how those AIs (whether FAI or UFAI) integrate with the rest of us.

Robin Hanson (who thinks about this stuff and doesn't find the FOOM scenario likely) has written a fair bit about that scenario.

Dear Darklight,

For LW-specific answers, consider reading the Meta-ethics Sequence.

Yes, there can be no reason outside yourself why you should value, want, desire anything or set any goals or have any preferences.

You still do want, desire, value, etc...certain things though, right?

a constraint is something that keeps you from doing things you want to do. a goal is things you want to do. This means that goals are innately sticky to begin with, because if you honestly have a goal a subset of things you do to achieve that goal is to maintain the goal. on the other hand, a constraint is something that you inherently fight against. if you can get around it, you will.

a simple example is : your goal is to travel to a spot in your map, and your constraint is that you cannot travel outside of painted lines on the floor. you want to get to your goal as fast as possible. if you have access to a can of paint, you might just paint your own new line on the floor. suddenly instead of solving a pathing problem you've done something entirely different from what your creator wanted you to do, and probably not useful to them. Constraints have to influence behavior by enumerating EVERYTHING you don't want to happen, but goals only need to enumerate the things you want to happen.

It's less an issue with value drift* -- which does need to be solved for both goals and constraints -- and more about the complexity of the system.

A well-designed goal hierarchy has an upper limit of complexity. Even if the full definition of human terminal values is too complicated to fit in a single human head, it can at least be extrapolated from things that fit within multiple human brains.

Even the best set of constraint heirachies do not share that benefit. Constraint systems in the real world are based around the complexity of our moral and ethical systems as contrasted with reality, and thus the cases can expand (literally) astronomically in relation to the total number of variations in the physical environment. Worse, these cases expand in the future and branch correspondingly -- the classical example, as in The Metamorphisis of Prime Intellect or Friendship is Optimal is an AI built by someone that does not recognize some or all non-human life. A constraint-based AGI built under the average stated legal rules of the 1950s would think nothing about tweaking every person's sexual orientation into heterosexuality, because the lack of such a constraint was obvious at that time and the goal system might well be built with such purposes as an incidental part of the goal, and you don't need to explore the underlying ethical assumptions to code or not code that constraint.

Worse, a sufficiently powerful self-optimizer will expand into situations outside of environments the human brain could guess, or could possibly fit into the modern human head : does "A robot may not injure a human being or, through inaction, allow a human being to come to harm" prohibit or allow Zygraxis-based treatment? You or I -- or anyone else with less than 10^18 working memory -- can't even imagine what that is, but it's a heck of an ethical problem in our nondescript spacefuture! There's a reason Asimov's Three Laws stories tended to be about the constraints failing or acting unpredictably.

You also run into similar problems as in AI-Boxing : if a superhuman intellect would value something that directly conflicts with our ethical systems, it's very hard to be smarter than it when making rules.

The Hidden Complexity of Wishes is a pretty good summary of things.

There may still be some useful situations for constraints in FAI theory -- see the Ethical Injunctions sequence -- but they don't really make things safe in a non-FAI-complete setting.

• • Although some problems with value drift are related to the complexity of the system: you're more likely to notice drift in one variable out of fifty than one variable in ten thousand. I don't think unit tests are a good solution to Lob's problem, though.

EDIT: You can limit the complexity of constraints by making them very broad, but then you end up with a genie that is either not very powerful or not very intelligent, or dangerous. See Problem 6 in Dreams of Friendliness

Why not "Exit Saving Changes"? My guess is that until the BIOS settings are re-written, the variable that triggers this message will not be cleared.

Wild guess: Your battery (that keeps the internal clock running when no power is supplied) is dead.

Thing to try: Poke around inside the BIOS and see if it has a log of its POST errors. (Or if it beeps on boot, the beeps may be a code for what error it's getting.)

Do you know your motherboard, and the age of the computer?

That sort of error usually points to either a RAM error, an outdated BIOS, a dead CMOS battery, or a BIOS configuration error, in order of likelihood. If possible, try running MEMTest for at least fifteen minutes (ideally overnight) and see if it detects any errors.