Stupid Questions Open Thread Round 4

At the moment, LW has provided negative benefit to my life. I recently quit my job to start learning positive psychology. My initial goal was to blog about positive psychology, and eventually use my blog as a platform to sell a book.

LW has made me deeply uncertain of the accuracy of the research I read, the words I write on my blog, and the advice I am writing in the book I intend to sell. Long-term, the uncertainty will probably help me by making me more knowledgeable than my peers, but in the short-term, demotivates (e.g. if I was sure what I was learning was correct, I would enthusiastically proselytize, which is a much more effective blogging strategy).

Still, I read on, because I've passed the point of ignorance.

Absence of evidence is evidence of absence :-) Most of us don't seem to get such benefits from reading LW, so learning about an individual case of benefit shouldn't influence your decisions much. It will probably be for spurious reasons anyway. Not sure about the camps, but my hopes aren't high.

Can anyone attest to getting real instrumental rationality benefits from reading Wikipedia? (As a control question; everyone seems to think that Wikipedia is obviously useful and beneficial, so is anyone getting "real instrumental rationality benefits" from it?)

I suspect that the "success equation", as it were, is something like expected_success = drive intelligence rationality, and for most people the limiting factor is drive, or maybe intelligence. Also, I suspect that changes in your "success equation" parameters take can take years to manifest as substantial levels of success, where people regard you as "successful" and not just "promising". And I don't think anyone is going to respond to a question like this with "reading Less Wrong made me more promising" because that would be dopey, so there's an absence of data. (And promising folks may also surf the internet, and LW, less.)

It's worth differentiating between these two questions, IMO: "does reading LW foster mental habits that make you better at figuring out what's true?" and "does being better at figuring out what's true make you significantly more successful?" I tend to assign more credence to the first than the second.

I was at July rationality minicamp, and in addition to many "epiphanies", one idea that seems to work for me is this, very simplified -- forget the mysterious "willpower" and use self-reductionism, instead of speaking in far mode what you should and want to do, observe in near mode the little (irrational) causes that really make you do things. Then design your environment to contain more of those causes which make you do things you want to do. And then, if the theory is correct, you find yourself doing more of what you want to do, without having to suffer the internal conflict traditionally called "willpower".

Today it's almost one month since the minicamp, and here are the results so far. I list the areas where I wanted to improve myself, and assign a score from 0 to 5, where 5 means "works like a miracle; awesome" and 0 means "no change at all". (I started to work on all these goals in parallel, which may be a good or bad idea. Bad part is, there is probably no chance succeeding in all at once. Good part is, if there is success in any part, then there is a success.)

• (5) avoiding sugar and soda
• (4) sleeping regularly, avoiding sleep deprivation
• (2) spending less time procrastinating online
• (2) exercising regularly
• (2) going to sleep early, waking up early
• (1) following my long-term plans
• (1) spending more time with friends
• (1) being organized, planning, self-reflecting
• (0) writing on blog, improving web page
• (0) learning a new language
• (0) being more successful at work
• (0) improving social skills and expanding comfort zone
• (0) spending more time outside

So far it seems like a benefit, although of course I would be more happy with greater/faster improvement. The mere fact that I'm measuring (not very exactly) my progress is suprising enough. I'm curious about the long-term trends: will those changes gradually increase (as parts of my life get fixed and turn to habit) or decrease (as happened with my previous attempts at self-improvement)? Expect a more detailed report in the end of December 2012.

How exactly did I achieve this? (Note: This is strongly tailored to my personality, it may not work for other people.) Gamification -- I have designed a set of rules that allow me to get "points" during the days. There are points e.g. for: having enough sleep, having an afternoon nap, meeting a friend, exercising (a specific amount), publishing a blog article, spending a day without consuming sugar, spending a day without browsing web, etc. Most of these rules allow only one point of given type per day, to avoid failure modes like "I don't feel like getting this point now, but I can get two of these points tomorrow". So each day I collect my earned points, literally small squares of colored paper (this makes them feel more real), and glue them on a paper form, which is always on my desk, and provides me a quick visual feedback on how "good" the previous days were. It's like a computer game (exact rules, quick visual feedback), which is exactly why I like it.

This specific form of gamification was not literally taught at Minicamp, and I was considering something similar years before. Yet I never did it; mostly because I was stopped by my monkey-tribe-belonging instincts. Doing something that other people don't do is weird. I tried to convince some friends to join me in doing this, but all my attempts failed; now I guess it's because admitting that you need some kind of help is low status, while speaking about willpower in far mode is high status. Being with people at Minicamp messed with my tribe instincts; meeting a community of people with social norm of doing "weird" things reduced my elephant's opposition to doing a weird thing. Sigh, I'm just a monkey, and I'm scared of doing things that other monkeys never do; even if it means being rational or winning.

I was a July minicamp attendee. I did the big reading through the Sequences thing when lukeprog was doing it at Common Sense Atheism, so I'd day fewer of the benefits were rationality level-ups and more were life hacking. Post-minicamp I am:

• doing sit-ups, push-ups, and squats every day (using the apps from the 200 situps guy), up from not doing this at all
• martial arts training four times a week (aikido and krav) again, up from not doing things at all
• using RTM to manage tasks which means
• dropping way fewer small tasks
• breaking tasks down into steps more efficiently
• knocked off about three lagging tasks (not timebound, so I was making no progress on them) in the month that I got back
• stopped using inbox as task manager, so I could actually only keep emails I was replying to in there
• using beeminder to get down to inbox zero (currently three)
• working in pomodoros has sped up my writing to the point where:
• I miss doing a daily posts to my blog more rarely (one over two weeks compared to 0-2 a week) and have had more double post days than previously (which translates into higher page views and more money for me)
• Less time writing left me more time for leisure reading

This topic has been raised dozens of times before, but the stories are scattered. Here's a sampling:

• Louie's What I've Learned from Less Wrong
• cousin_it on how LW helps him notice bullshit
• FrankAdamek on gains from LW
• cata's group rationality diary thread contains lots of stories of people benefiting from applying the lessons learned in rationality camps
• A couple people have posted about how LW deconverted them from their religions, but I can't recall where

But also see this comment from Carl Shulman.

July minicamper here. My own life has had enough variance in the past few months over many variables (location, job, romantic status) with too many exogenous variables for me to be very confident about the effect of minicamp, aside from a few things (far fewer totally wasted days than I used to suffer from what I saw as being inescapably moody).

But I've gained an identifiable superpower in the realm of talking helpfully to other people by modeling their internal conflicts more accurately, by steering them toward "making deals with themselves" rather than ridiculous memes like "using willpower", and by noticing confusion and getting to the root of it via brainstorming and thought experiments. And the results have absolutely floored people, in three different cases. If you're worried about epiphany addiction, then I suppose you might label me a "carrier" (although there's the anomalous fact that friends have followed through on my advice to them after talking to me).

I think the probability of my having got my current job without LW etc. is under 20%.

Subjectively I feel happier and more effective, but there's not reliable external evidence for this. I've gotten better at talking to people and interacting in positive ways thanks to using metacognition, and my views have become more consistent. Timeless thinking has helped me adopt a diet and stick to it, as well as made me start wearing my seatbelt.

Maybe the easiest way to understand UDT and TDT is:

• UDT = EDT without updating on sensory inputs, with "actions" to be understood as logical facts about the agent's outputs
• TDT = CDT with "causality" to be understood as Pearl's notion of causality plus additional arrows for logical correlations

Comparing UDT and TDT directly, the main differences seem to be that UDT does not do Bayesian updating on sensory inputs and does not make use of causality. There seems to be general agreement that Bayesian updating on sensory inputs is wrong in a number of situations, but disagreement and/or confusion about whether we need causality. Gary Drescher put it this way:

Plus, if you did have a general math-counterfactual-solving module, why would you relegate it to the logical-dependency-finding subproblem in TDT, and then return to the original factored causal graph? Instead, why not cast the whole problem as a mathematical abstraction, and then directly ask your math-counterfactual-solving module whether, say, (Platonic) C's one-boxing counterfactually entails (Platonic) $1M? (Then do the argmax over the respective math-counterfactual consequences of C's candidate outputs.)

(Eliezer didn't give an answer. ETA: He did answer a related question here.)

I'm not an expert but I think this is how it works:

Both decision theories (TDT and UDT) work by imagining the problem from the point of view of themselves before the problem started. They then think "From this point of view, which sequence of decisions would be the best one?", and then they follow that sequence of decisions. The difference is in how they react to randomness in the environment. When the algorithm is run, the agent is already midway through the problem, and so might have some knowledge that it didn't have at the start of the problem (e.g. whether a coinflip came up heads or tails). When visualising themselves at the start of the problem TDT assumes they have this knowledge, UDT assumes they don't.

An example is Counterfactual Mugging:

Imagine that one day, Omega comes to you and says that it has just tossed a fair coin, and given that the coin came up tails, it decided to ask you to give it $100. Whatever you do in this situation, nothing else will happen differently in reality as a result. Naturally you don't want to give up your $100. But see, the Omega tells you that if the coin came up heads instead of tails, it'd give you $10000, but only if you'd agree to give it $100 if the coin came up tails.

TDT visualises itself before the problem started, knowing that the coin the coin will come up tails. From this point of view the kind of agent that does well is the kind that refuses to give $100, and so that's what TDT does.

UDT visualises itself before the problem started, and pretends it doesn't know what the coin does. From this point of view the kind of agent that does well is the kind that gives $100 in the case of tails, so that's what UDT does.

Follow-up: Is it in how they compute conditional probabilities in the decision algorithm? As I understand it, that's how CDT and EDT and TDT differ.

There are some but not lots of "writings" produced internally by SingInst that are not available to the public. There's lots of scribbled notes and half-finished models and thoughts in brains and stuff like that. We're working to push them out into written form, but that takes time, money, and people — and we're short on all three.

The other problem is that to talk about strategy we first have to explain lots of things that are basic (to a veteran like you but not to most interested parties) in clear, well-organized language for the first time, since much of this hasn't been done yet (these SI papers definitely help, though: 1, 2, 3, 4, 5, 6). To solve this problem we are (1) adding/improving lots of articles on the LW wiki like you suggested a while back (you'll see a report on what we did, later), and (2) working on the AI risk wiki (we're creating the map of articles right now). Once those resources are available it will be easier to speak clearly in public about strategic issues.

We hit a temporary delay in pushing out strategy stuff at SI because two of our most knowledgable researchers & strategists became unavailable for different reasons: Anna took over launching CFAR and Carl took an extended (unpaid) leave of absence to take care of some non-SI things. Also, I haven't been able to continue my own AI risk strategy series due to other priorities, and because I got to the point where it was going to be a lot of work to continue that sequence if I didn't already have clear, well-organized write-ups of lots of standard material. (So, it'll be easier for me to continue once the LW wiki has been improved and once the AI risk wiki exists, both of which we've got people working on right now.)

Moreover, there are several papers in the works — mostly by Kaj (who is now a staff researcher), with some help from myself — but you won't see them for a while. You did see this and this, however. Those are the product of months of part-time work from several remote researchers, plus analysis by Kaj and Stuart. Remote researchers are currently doing large lit reviews for other paper projects (at the pace of part-time work), but we aren't at the stage of analyzing those large data sets yet so that we can write papers about what we found.

Also, a lot of work currently scattered around in papers and posts by SI and FHI people is being collected and tightly organized in Nick Bostrom's forthcoming monograph on superintelligence, which will be a few hundred pages entirely about singularity strategy. Having much of "the basics" organized in that way will also make it easier to produce additional strategy work.

I love working on problems like these. If the specifics of the "circles" are written down anywhere, or if you care to describe them, I'd happily give it a whack. I won't claim to be an expert, but I enjoy complex problem solving tasks like this one too much not to offer.

You could always choose to manage ignorance by choosing a prior. It's not obvious whether you should. But as it turns out, we have results like the complete class theorem, which imply that EU maximization with respect to an appropriate prior is the only "Pareto efficient" decision procedure (any other decision can be changed so as to achieve a higher reward in every possible world).

This analysis breaks down in the presence of computational limitations; in that case it's not clear that a "rational" agent should have even an implicit representation of a distribution over possible worlds (such a distribution may be prohibitively expensive to reason about, much less integrate exactly over), so maybe a rational agent should invoke some decision rule other than EU maximization.

The situation is sort of analogous to defining a social welfare function. One approach is to take a VNM utility function for each individual and then maximize total utility. At face value it's not obvious if this is the right thing to do--choosing an exchange rate between person A's preferences and person B's preferences feels pretty arbitrary and potentially destructive (just like choosing prior odds between possible world A and possible world B). But as it turns out, if you do anything else then you could have been better off by picking some particular exchange rate and using it consistently (again, modulo practical limitations).

What AlexMennen said. For a Bayesian there's no difference in principle between ignorance and risk.

One wrinkle is that even Bayesians shouldn't have prior probabilities for everything, because if you assign a prior probability to something that could indirectly depend on your decision, you might lose out.

A good example is the absent-minded driver problem. While driving home from work, you pass two identical-looking intersections. At the first one you're supposed to go straight, at the second one you're supposed to turn. If you do everything correctly, you get utility 4. If you goof and turn at the first intersection, you never arrive at the second one, and get utility 0. If you goof and go straight at the second, you get utility 1. Unfortunately, by the time you get to the second one, you forget whether you'd already been at the first, which means at both intersections you're uncertain about your location.

If you treat your uncertainty about location as a probability and choose the Bayesian-optimal action, you'll get demonstrably worse results than if you'd planned your actions in advance or used UDT. The reason, as pointed out by taw and pengvado, is that your probability of arriving at the second intersection depends on your decision to go straight or turn at the first one, so treating it as unchangeable leads to weird errors.

So if you're a Bayesian decision-maker, doesn't that mean that you only ever face decisions under risk, because at they very least you're assigning ignorance priors to the outcomes for which you're not sure how to assign probabilities?

Correct. A Bayesian always has a probability distribution over possible states of the world, and so cannot face a decision under ignorance as you define it. Coming up with good priors is hard, but to be a Bayesian, you need a prior.

Bayesian decisions cannot be made under an inability to assign a probability distribution to the outcomes.

As mentioned, you can consider a Bayesian probability distribution of what the correct distributions will be; if you have no reason to say which state, if any, is more probable, then they have the same meta-distribution as each other: If you know that a coin is unfair, but have no information about which way it is biased, then you should divide the first bet evenly between heads and tails, (assuming logarithmic payoffs).

It might make sense to consider the Probability distribution of the fairness of the coin as a graph: the X axis, from 0-1 being the chance of each flip coming up heads, and the Y axis being the odds that the coin has that particular property; because of our prior information, there is a removable discontinuity at x=1/2. Initially, the graph is flat, but after the first flip it changes: if it came up tails, the odds of a two-headed coin are now 0, the odds of a .9999% heads coin are infinitesimal, and the odds of a tail-weighted coin are significantly greater: Having no prior information on how weighted the coin is, you could assume that all weightings (except fair) are equally likely. After the second flip, however, you have information about what the bias of the coin was- but no information about whether the bias of the coin is time-variable, such that it is always heads on prime flips, and always tails on composite flips.

If you consider that the coin could be rigged to a sequence equally likely as that the result of the flip could be randomly determined each time, then you have a problem. No information can update some specific lacks of a prior probability.

This reminds me of a recent tangent on Kelly betting. Apparently it's claimed that the unusalness of this optimum betting strategy shows that you should treat risk and ignorance differently - but of course the difference between the two situations is entirely accounted for by two different conditional probability distributions. So you can sort of think of situations (that is, the probability distribution describing possible outcomes) as "risk-like" or "ignorance-like."

I think cryonics is more likely to be a mistake than a scam, but that might just be my general belief that incompetence is much more common than malice.

I think there is a very good chance some cryonics organizations are in fact scams.

Has there been work done or discussion about the friendliness of Ems / Uploads?

• http://lesswrong.com/lw/66n/resetting_gandhieinstein/
• http://lesswrong.com/r/discussion/lw/5jb/link_whole_brain_emulation_and_the_evolution_of/

As soon as the first upload is successful then patient zero will realize he's got unimaginable (brain)power, will start talking in ALL CAPS, and go FOOM on the world, bad end. For the sake of argument, lets say we get lucky and first upload is incredibly nice, and just wants to help people. Eventually the second, or the third, or the twenty fifth upload decides to FOOM over everybody. It's still bad end.

Why can't the first upload FOOM, but in a nice way?

That strikes me as a problem that's just as hard as FAI. There seems like no way to solve it that doesn't involve a friendly AGI controlling the upload world.

Some people suggest uploads only as a stepping stone to FAI. But if you read Carl's paper (linked above) there are also ideas for how to create stable superorganisms out of uploads that can potentially solve your regulation problem.

Ems can still talk to one another and convince one another of things. How do we know they won't all be hijacked by meme-viruses, and transformed Agent Smith style?

This one is a little silly. Humans get hijacked by meme-viruses as well, all the time; it does cause problems, but mostly other humans manage to keep them line.

But as for the rest, yes, I agree with you that an upload scenario would have huge risks as well. Not to mention the fact that there might be a considerable pressure towards uploads merging together and ceasing to be individuals in any meaningful sense of the term. Humanity's future seems pretty hopeless to me.

Human uploads have been discussed as dangerous. But a friendly AI is viewed as an easier goal than a friendly upload, because an AI can be designed.

As soon as the first upload is successful then patient zero will realize he's got unimaginable (brain)power, will start talking in ALL CAPS, and go FOOM on the world, bad end.

Now, I have to admit I'm not too familiar with the local discourse re:uploading, but if a functional upload requires emulation down to individual ion channels (PSICS-level) and the chemical environment, I find it hard to believe we'll have the computer power to do that, a million times faster, and in a volume of space small enough that we don't have to put it under a constant waterfall of liquid Helium.

I don't expect femtotechnology or rod logic any time soon, the former may not even be possible at all and the latter is based on some dubious math from Nanosystems; so where does that leave us in terms of computing power? (Assuming, of course, that Clarke's law is a wish-fulfilling fantasy). I understand the reach of Bremermann's Limit, but it may not be possible to reach it, or there may be areas in between zero and the Limit that are unreachable for lack of a physical substrate for them.

Ems have similar human psychology, with adds. I presume they can't escalate well as AIs, even in coalescence cases.

Possible dangers come with the same dangers of now but with more structural changes. If they have artificial agents in they realm, some cheap nanotech, etc. Conflicts has costs too.

If X is just a binary proposition that can be true or false once and for all, and A and B have arrived at the same degree-of-belief, they are equally confident. A has updated on evidence related to X since childhood, and found that it's perfectly balanced in either direction. The only way A can be said to be "more confident" than B is that A has seen a lot of evidence already, so she won't update her conclusion upon seeing the same evidence again; on the other hand, all evidence is new to B.

Things get more interesting if X is some sort of random variable. Let's say we have a bag of black and white marbles. A has seen people draw from the bag 100 times, and 50 of them ended up with white marbles. B only knows the general idea. Now, both of them expect a white marble to come up with 50% probability. But actually, they each have a probability distribution on the fraction of white marbles in the bag. The mean is 1/2 for both of them, but the distribution is flat for B, and has a sharp peak at 1/2 for A. This is what determines how confident they are. If C comes along and says "well, I drew a white marble", then B will update to a new distribution, with mean 2/3, but A's distribution will barely shift at all.

Nick Bostrom in Infinite Ethics terms this "the causal approach" to the problem of infinities, and comments:

An advocate for the causal approach might point out that, according to relativity theory, nobody can influence events outside their future light cone. Cosmology suggests that the number of value-bearing locations (such as lives, or seconds of consciousness etc.) in our future light cone is finite. Given our best current physics, therefore, the causal approach appears to avoid paralysis.

Not so fast. Basing our ethics on an empirical fact about the laws of nature means that it cannot satisfy the highest methodological standard (cf. section 1). Well, we might be able to live with that. But the situation is much worse: the causal approach fails even in the situation we are actually in, thus failing to meet even the lowest possible acceptability criterion for a moral theory. This is because reasonable agents might— in fact, should—assign a finite non-zero probability to relativity theory and contemporary cosmology being wrong. When a finite positive probability is assigned to scenarios in which it is possible for us to exert a causal effect on an infinite number of value-bearing locations (in such a way that there is a real number r>0 such that we change the value of each of these location by at least r), then the expectation value of the causal changes that we can make is undefined.31 Paralysis will thus strike even when the domain of aggregation is restricted to our causal sphere of influence.

We could attempt to avoid this problem arising from our subjective uncertainty about the correctness of current physics by stipulating that the domain of aggregation should be restricted to our future light cone even if, contrary to special relativity, we could causally affect locations outside it. With this stipulation, we could ignore the physically far-fetched scenarios in which faster-than-light influencing is possible.

This tweak is not as good as it may appear. If, contrary to what current physics leads us to believe, it is in fact possible for us (or for somebody else, perhaps a technologically more advanced civilization) to causally influence events outside our (their) future light cone, moral considerations would still apply to such influencing. According to the present proposal, we should not factor in such considerations even if we thought superluminal propagation of causal influence to be quite likely; and that is surely wrong.

Moreover, even if the propagation of our causal effects is limited by the speed of light, it could still be possible for us to influence an infinite number of locations. This could happen, for instance, in a spatially infinite cyclic spacetime or in a steady-state cosmology.

...though this might not be relevant to Eliezer's actual reasons to reject total utilitarianism, because infinite ethics a la Bostrom would make average utilitarianism just as infeasible:

The threat is not limited to hedonistic utilitarianism. Utilitarian theories that have a broader conception of the good—happiness, preference-satisfaction, virtue, beauty-appreciation, or some objective list of ingredients that make for a good life—face the same problem. So, too, does average utilitarianism, mixed total/average utilitarianism, and prioritarian views that place a premium on the well-being of the worst off. In a canonically infinite world, average utility and most weighted utility measures are just as imperturbable by human agency as is the simple sum of utility.

The domain of a utility function is possible states of the world. The whole world, not just the parts you can physically affect. Some utility functions (such as total utilitarianism) can be factored into an integral over spacetime (and over other stuff for Tegmark IV) of some locally-supported function, and some can't. If you have a non-factorable utility function, then even if the world is partitioned into non-interacting pieces x and y and you're in x, the value of y still affects ∂U/∂x, and is thus relevant to decisions.

The board game "Wits and Wagers" might qualify for what you are looking for. Game play is roughly as follows: A trivia question is asked and the answer is always a number (e.g., "How many cups of coffee does the average American drink each year?", "How wide, in feet, is an American football field?"). All the players write their estimate on a slip of paper and then then they are arranged in numerical order on the board. Everybody then places a bet on the estimate they like the best (it doesn't have to be your own). The estimates near the middle have a low payback (1:1, 2:1) and the estimates near the outside have a larger payback (4:1). If your estimate is closest to the actual number or if you bet on that one, will get a payback on your bet.

Zendo, Nomic, Eleusis, Master Mind ... and yeah, probably Poker for probability.

Petals around the rose: http://www.borrett.id.au/computing/petals-j.htm

I asked a similarish question here: http://stats.stackexchange.com/questions/28925/good-games-for-learning-statistical-thinking - for example, these games for simple stats.

Settlers of Catan isn't a rationality game, but it's great for teaching economics. I play Catan with my little brothers and it has helped them understand concepts like comparative advantage, supply and demand, cartels, opportunity cost, time value of money, and derivatives markets. Just make sure you play with everyone showing what resource cards they have, instead of keeping their resource cards hidden. More interesting trades that way.

In the discussion about AI-based vs. upload-based singularities, and the expected utility of pushing for WBE (whole-brain emulation) first, has it been taken into account that an unfriendly AI is unlikely to do something worse than wiping out humanity, while the same isn't necessarily true in an upload-based singularity?

"Yes" in the sense that people are aware of the argument, which goes back at least as far as Vernor Vinge, 1993, but "no" in the sense that there are also arguments that it may not be highly unlikely that a failed attempt at FAI will be worse than extinction (especially since some of the FAI proposals, such as Paul Christiano's, are actually very closely related to uploading), and also "no" in the sense that we don't know how to take into account considerations like this one except by using our intuitive judgments which seem extremely unreliable.

I've asked this several times before. As far as I can make out, no (published) text answers this question. (If I'm wrong I am very interested in learning about it.)

The CEV doc assumes without any proof, not just that we (or a superintelligent FAI) will find a reconciling strategy for CEV, but that such a strategy exists to be found. It assumes that there is a unique such strategy that can be defined in some way that everyone could agree about. This seems to either invite a recursion (everyone does not agree about metaethics, CEV is needed to resolve this, but we don't agree about the CEV algorithm or inputs); or else to involve moral realism.

Individuals have Volitions and (hopefully) Extrapolatable Volitions. If many people have EVs that 'agree', they interfere constructively (like waves), and that becomes part of the group's Coherant Extrapolated Volition. If they 'disagree' on some issue, they interfere distructively, and CEV has nothing to say on the issue.

(I'd be nice to be able to explain this by saying that individuals have EVs but not CEVs, excelent clearly they have a degenerate case of CEV if they have an EV)

What happens if my CEV is different than yours?

Then the "coherent" qualifier does not apply, does it? Are you asking how to construct CEV from the multitude of PEVs (P for personal)?

Does the FAI put us all in our own private boxes

Presumably those folks whose PEV does not mind boxing will get boxed, and the rest will have to be reconciled into the CEV, if possible. Or maybe commensurate PEVs get boxed together into partial CEV worlds.

The hard part is what to do with those whose PEV is incompatible with other people having different ideas from theirs. Eh, maybe not that hard. Trickery or termination is always an option when nothing better is available.

I don't understand what you mean. Can you give me an example of preferences that are intransitive but not cyclic?

Just in case - synchronising the definitions.

I usually consider something transitive if "X≥Y, Y≥Z then X≥Z" holds for all X,Y,Z.

If this holds, preferences are transitive. Otherwise, there are some X,Y,Z: X≥Y, Y≥Z, Z>X. I would call that cyclical.

We wouldn't. However, the FAI knows that if it changed its code to unFriendly code, then unFriendly things would happen. It's Friendly, so it doesn't want unFriendly things to happen, so it doesn't want to change its code in such a way as to cause those things - so a proper FAI is stably Friendly. Unfortunately, this works both ways: an AI that wants something else will want to keep wanting it, and will resist attempts to change what it wants.

There's more on this in Omohundro's paper "Basic AI Drives"; relevant keyword is "goal distortion". You can also check out various uses of the classic example of giving Gandhi a pill that would, if taken, make him want to murder people. (Hint: he does not take it, 'cause he doesn't want people to get murdered.)

I don't think Aumann's agreement theorem is the problem here.

If QI is true, the only outcome I can observe is surviving.

What does it mean for QI to be true or false? What would you expect to happen differently? Certainly, whether or not QI is true, the only outcome you can observe is surviving, so I don't see how you're updating your belief.

As far as I understand, there is still no satisfactory theory that would include both quantum mechanics and general relativity (i.e. the possibility for spacetime not to be same).

I would expect that in unified theory spacetime structure would be a part of the state undergoing quantum superposition.

The question makes sense, but the answers probably won't.

Questions like this are usually approached in an upside-down way. People assume, as you are doing, that reality is "just neurons" or "just atoms" or "just information", then they imagine that what they are experiencing is somehow "just that", and then they try to live with that belief. They will even construct odd ways of speaking, in which elements of the supposed "objective reality" are substituted for subjective or mentalistic terms, in order to affirm the belief.

You're noticing that "self-awareness" or "the feeling that something is happening" or "the feeling that I exist" doesn't feel like it's the same thing as "neurons"; though perhaps you will tell yourself - as Wittgenstein may have done - that you don't actually know what being a pack of neurons should feel like, so how do you know that it wouldn't feel exactly like this?

But if you pay attention to the subjective component of your thought, even when you're thinking objectively or scientifically, you'll notice that the reduction actually goes in the other direction. You don't have any direct evidence of the "objective existence" of neurons or atoms or "information". The part of reality that you do know about is always some "experience" that is "happening", which may include thoughts about an objective world, that match up in some way with elements of the experience. In other words, you don't know that there are neurons or atoms, but you can know that you are having thoughts about these hypothetical objects. If you're really good at observing and analyzing your thoughts, you may even be able to say a lot about the conscious mental activity which goes into making the thought and applying it to experience.

The fundamental problem is that the physical concept of reality is obtained by taking these conscious states and amputating the subjective part, leaving only the "object" end. Clearly there is a sense in which the conscious subject is itself an "object", it's something that exists. But all the dimly apprehended ontological peculiarities which make consciousness what it is, and which give rise to these nebulous names like intentionality and qualia, are not part of the ontology of "objects". If you think about another person, and think of them as possessing a consciousness just as you do, then your object-of-thought is still ontologically a subject, but of course that's not what we do in physics. We just focus on a few attributes which we have learnt to think about rigorously, like space, time, quantity, and then we assume that they are the whole of reality.

That is the essence of the upside-down approach to the problem of physics and consciousness. You have to go in the other direction, acknowledge that consciousness is what it is, and then try to understand physics so that something in physics has all those properties. This isn't easy and it's why I promote certain kinds of quantum brain theories, because quantum mechanics, under certain interpretations, can contain complex "wholes" that might be consciousness. But even if that were true, you would still have one more challenge: find a way to think of the usual mathematical ontology of physics as the superficial functional description, and the lived experience of existing as a glimpse of the true reality, rather than the other way around.

The general rule seems to be "if your post is interesting and well-written enough, it's fine"; hard to say anything more specific than that. No strong norms about length (even quite short posts have been heavily upvoted), and formalism is optional, but a post that's heavy on math or formal logic will probably get less readers.

It would seem to me that biological restrictions on space travel wouldn't apply to a sufficiently advanced AI. This AI could colonize other worlds using near speed of light travel with minimal physical payload and harvest the raw materials on some new planet using algorithms programmed in small harvesting bots.

Pretty much right.

If this is possible then it seem to me that unfriendly AI might not be that much of a threat since they would have many more "habitable" worlds to harvest/live on (like Venus or Mars, comets, asteroids, or extra-solar planets).

We would eventually like to inhabit the currently uninhabitable planets. Terraforming, self modification, sealed colonies, or some combination of those will eventually make this feasable. At that time, we would rather that those planets not fight back.

Symmetrically, an unfriendly process will not be satisfied with taking merely Mercury, Venus, Mars, Jupiter, Saturn, Uranus, Neptune, and the rest of the universe; it will want to do its thing on earth as well. The choice between "kill the humans and take over earth" and "don't kill the humans and don't take over earth" is independent of the existence of other territory, so it doesn't matter and it will kill us.

(the short answer is that there is no "satisfied" or "enough" among nonhuman agents.)

Another thing. If this is possible it sort of leads to a paradox: Why hasn't it happened already with other intelligent life on other planets?

You mean the fermi paradox? You'll have to expand, but note that a singularity will expand at lightspeed (=we wouldn't see it until it were here), and it will consume all resources (= if it had been here, we wouldn't).

That topic is full of a lot of confusion and everyone seems to have different intuitions.

I for one am not proper utilitarian because it seems unnaturally simple (when we have no reason to suspect that human value should be simple).

But an additional awesome person seems like a good thing, if it doesn't make the world suck more.

I am (as much as I can) vigorously trying to make lots of money to fund positive-singularity research.

Sounds like Socrates.