Posts
Comments
I agree that deriving morality from stated human values is MUCH more ethically questionable than deriving it from human values, stated or not, and suggest that it is also more likely to converge. This creates a probable difficulty for CEV.
It seems to me that if it's worth destroying Huygens to stop the Superhappies it's plausibly worth destroying Earth instead to fragment humanity so that some branch experiences an infinite future so long as fragmentation frequency exceeds first contact frequency. Without mankind fragmented, the normal ending seems inevitable with some future alien race. Shut-up-and-multiply logic returns error messages with infinite possible utilities, as Peter has formally shown, and in this case it's not even clear what should be multiplied.
Hmm. I remember being non-reflective in first grade but not in second grade. One consequence was that I couldn't re-write explicit beliefs in response to new information and I saw general injunctions and commands as relatively binding and automatic. Conflicting commands couldn't be accommodated, nor could common sense. I don't think that my emotions were any more intense. I never re-wrote myself, or noticed a change at the time, but I notice it in my memories. Early ones don't include the question "why am I doing this?" or "why do this rather than that". In 6th grade I suffered catastrophic failure to relate to anyone who was not reflective and largely but incompletely corrected this failure as a college sophomore at age 18. Efforts to correct it continue, with large steps in the last 2 years.
I agree with much of what Robin has said here but wish he would write his own blog post about it.
Actually RU, that's a good approximation for many/most professions, but not all that good an approximation.
http://www.vanderbilt.edu/Peabody/SMPY/DoingPsychScience2006.pdf
gives more detail, showing a significant marginal impact from, at the least, 99.99th percentile math achievement at age 12 relative to merely 99.8th percentile math achievement at age 12.
Phil: Your estimate rewards precision and penalizes self estimate of precision. A person of a given level of precision should be rewarded for believing their precision to be what it is, not for believing it to be low. If you had self-estimate of precision in the numerator that would negate Nick's claim, but then you could drop the term from both sides.
Eliezer: I'm pretty sure that MANY very smart people learn more from working on hard problems and failing quite frequently than from reading textbooks and practicing easy problems. Both should be part of an intellectual diet.
Maksym: We actually do need someone to translate all this OB stuff very badly, though maybe it's desirable to wait for the book. Still, someone should be presenting it. As for convincing smart college students, there are three fairly separate barriers here, those to rationality, those of information and those to action. I recommend working on barriers to rationality and action first and in conjunction, belief second, and let people find the info themselves. Politics is the natural subject to frame as rationality. Simply turn every conversation where politics comes up into an opportunity to discourse on OB. Rules of etiquette are weak at Harvey Mudd, so this should be OK.
Denis: In technical fields? If so, I unhesitatingly deny the data. I suggest you look at Gottfredson. Lynn is far from trustworthy, but may also be summarizing. Do you really think that people who can't pull 600 on the SAT Math can do engineering?
You pretty much said it. Hypotheses suggested by mind-projection priors turning out to be true pretty much refutes Occam and consequentially science.
It's not on that level, that's the level which I respond to with the forbidden bet, e.g. p = 0, along with all the other stuff that implies strongly that our concepts of probability are simply broken.
Reason is a mistake for less extreme reasons such as "I'm dreaming" or "I'm a Boltzman Brain" or some forms of "my life is not merely a simulation but a psychological experiment".
I took psi seriously back when I thought that the scientific method defined rationality. Once I learned about Bayes I realized that the sort of reports of psi that science turns up would be expected if psi isn't real while much more blatant things would be expected if real psi inspired the investigation. I also noticed that priors matter and psi really should be ignored without very large effects based on low priors. Somewhat earlier pre-Bayes psi had blended somewhat into the category "Everything you know is wrong" and loose specific identity as 'psi'. Post-Bayes the "Everything you know is wrong" itself split into a few categories and psi went in the "reason is a mistake" extreme category.
Thinking about the future as today + diff is another serious problem with similar roots.
Robin: Great Point! Eliezer: I'm awaiting that too.
Shane Legg: I don't generally like sf, film or otherwise, but try "Primer". Best movie ever made for <$6000 AND arguably best sf movie. The Truman Show was good too in the last decade or so. That's probably it though. Minority Report was OK.
Dennis Bider: A BASIC and ESSENTIAL though these days largely forgotten principle of liberal society is that it can be the case, and often is, that behavior X is NOT OK but that banning behavior X would also be NOT OK.
Pete, if you do that then being a casual decision theorist won't, you know, actually Win in the one shot case. Note that evolution doesn't produce organisms that cooperate in one shot prisoners dilemmas.
But Eliezer, you can't assume that Clippy uses the same decision making process that you do unless you know that you both unfold from the same program with different utility functions or something. If you have the code that unfolds into Clippy and Clippy has the code that unfolds into you it may be that you can look at Clippy's code and see that Clippy defects if his model of you defects regardless of what he does and cooperates if his model of you cooperates if your model of him cooperates, but you don't have his code. You can't say much about all possible minds or about all possible paperclip maximizing minds.
I would say that in my experience evangelicals and traditional Christians max out at in the low 140s IQ wise.
Nick: Do you imagine that they would tell you so? Also, you are a) young, and b) haven't been in any setting where people come from a large variety of social backgrounds.
Highly intelligent Christians, Dyson for instance, are likely to believe roughly the same things you do but frame them differently. Tegmark 4 = Spinoza's god, for instance.
Hopefully: You and may others. I will if I ever pull together the emotional resources to, which seem unusually high for me. It's very demanding of effort for me to address a large group, most of whom will fail to "get it" whatever I do.
Seconding Hopefully Anonymous on this point. Also emphasizing that not infrequently, when the accuracy of beliefs that a nerd can promote is low due to inferential distance or to gaps in intelligence, nerds tend to give true statements without bridging the inferential distance, predictably promoting less than maximally accurate beliefs, primarily, I would say, motivated by the self-righteous feeling they get from telling the truth and the feeling of superior righteous indignation they get from their perceived inferiors showing ignorance by not understanding. If you are an atheist and a Christian wants to know if you believe in God, there IS NO Bayesian level honest answer, but there never is when dealing with people who aren't at least ancient art rationalists. The most honest answer is "Yes", which will be interpreted as something like "I believe in not eating babies" or "I believe in expressing my loyalty towards people and groups I care about" or the like.
Seconding Scott Aaronson (especially on the tragedy of the US getting nukes too late) and Michael G R, though with the suggestion that the "demo" bomb could have been Trinity.
Eliezer: I think that the hypocrisy of the US is mostly in our maintaining a large arsenal but telling others that they can't have any, not in having used nukes. I don't see a case for .1% probability increase for nuclear war without using the bomb which is stronger than the case for the inverse, but I do see a million dead Japanese. Also, in terms of catastrophic risks, a couple more years of total war to encourage ultimately disastrous cultural changes in the US.
What I really want to know about WWII is why Hitler wasn't assassinated before it was too late. Surely if the situation with Archduke Ferdinand teaches us anything it's that Assassination can solve all of one's geo-political problems ;-(
Eliezer: I think that you misunderstand Roko, but that doesn't really matter, as he seems to understand you fairly well right now and to be learning effectively.
Unknown: Not at all. Utility maximization is very likely to lead to counterintuitive actions, and might even lead to humanly useless ones, but the particular actions it leads to are NOT whatever salient actions you wish to justify but are rather some very specific set of actions that have to be discovered. Seriously, you NEED to stop reasoning with rough verbal approximations of the math and actually USE the math instead. Agreement emerges from Bayes, but the mere fact that you call something agreement doesn't strongly suggest that it is Bayesian. Seriously I have tried to communicate with you, Carl has tried, and Nick has tried. You aren't interested in figuring out the actual counter-intuitive consequences of your beliefs, as you are too afraid that you would have to act, you know, counter-intuitively, which in actuality you don't do at all. As far as I can tell you aren't worth any of us wasting any more time on.
Lara: invertebrates are, I believe, generally believed to have MUCH simpler sets of neurotransmitters than vertebrates. Think how few genes fruit flies have.
Great comments thread! Thanks all!
Seconding Roko, Carl, HA, Nick T, etc.
Eliezer or Robin: Can you cite evidence for "we can more persuasively argue, for what we honestly believe". My impression is that it has been widely assumed in evolutionary psychology and fairly soundly refuted in the general psychology of deception, which tells us that the large majority of people detect lies at about chance and that similar effort seems to enable the development of the fairly rare skill of the detection of lies and evasion of such detection.
Carl: Unknown and Utilitarian could be distinct but highly correlated (we're both here after all). In principle we could see them as both unpacking the implications of some fairly simple algorithm. Have you noticed them both making the same set of mistakes in their efforts to understand Bayesian reasoning, anthropics, decision theory, etc? Still could be the same program running on different sets of wetware.
I second Hopefully on criticism of the strawman postmodernist. Honestly, I think that academic disciplines, or even schools, where everyone is completely full of it are extremely rare. There are thoughtful, intelligent, and honest people who frame important and fairly novel ideas in the terminology of sociology, academic feminism, Freudianism, behaviorism, even, math-help-us, Jung. Perfect intellectual honest and intelligence among humans are a chimera, and different disciplines aspire to different approximations thereof by establishing different sorts of standards for esteem and for publication.
As a general summary, I would say that the Enlightenment was, to a large extent, a set of proscriptions regarding what types of questions should and should not be asked, argumentative styles should and should not be used, and hand waves are and are not allowed. Some set of standards is necessary for functional discourse under imperfect honesty, and the Enlightenment standards enabled fruitful conversation among people of merely, perhaps, 70th percentile honesty and 96th percentile intelligence, an AMAZING standard that reshaped history. Sadly, each set of standards disables productive discussion of certain subject matter, and places one's beliefs on ultimately unsound foundations, so re-foundation is ultimately necessary. The more dysfunctional academic disciplines are largely those which try to deal, possibly prematurely, with subject matter for which we lack good honesty enforcing protocols, including subject matter foundational to the Enlightenment subject matter, but for individuals in those disciplines insights are indeed possible and low-hanging fruit are even abundant.
BTW, I have significant personal experience with one of "the smartest guys in the room" and yes, they were (or at least he is) VERY smart by any normal business world standards. He's particularly great at giving obvious in retrospect answers to marketing type problems. It would be a somewhat unusual room full of business people or other social elites where the guy I'm thinking of isn't the smartest guy.
Caledonian: I can't think of anyone EVER choosing to interpret statements as stupid rather than sensible to the degree to which you do on this blog. There is usually NO ambiguity and you still get things wrong and then blame them for being stupid.
In all honesty why do you post here? On your own blog you are articulate and intelligent. Why not stick with that and leave commenting to people who want to actually respond to what people say rather than to straw men?
Laura: Should I have explained it in terms of your being Jewish? Geometry is more about logic and symbols, what the WAIT calls "Verbal" intelligence, than it is about visuospatial intelligence (which may be related to why its on the SAT). Does anyone think that its significantly less common for women to be competent sketch artists than for men to be? That honestly never occurred to me. But hey, you are a med student; how good are you at identifying expected drug-ligand interactions based on molecular shapes compared to similarly trained men (preferably gentile men) who are your approximate equals (have you met any?) at anagrams or at writing papers? If the answer is "quite, thank you" then I was wrong in my attribution to you of strong verbal relative to visuospatial skills. You know your abilities better than I do. Even in that case I think my comment above still stands. It just isn't an insult to claim that two people differ in their relative strengths, at least so long as those strengths are socially held in approximately equal regard. Why should this be different for groups of people?
I definitely do think that it could be an insult to claim that people or groups differ in relative strengths if the strengths in question are NOT held in similar regard, and I am aware that this is a real tendency. It is also a real and unfortunate fact that the same strengths can be socially held in high regard when held by members of one gender (almost always male) and in low regard in the other gender (almost always female). Really though, that is not plausibly what is going on here. Verbal abilities are held in MUCH higher regard in our society than visuospatial abilities. If anything, the visuospatial tasks are included in non-military IQ tests partly as a habitual carry-over from a time when visuospatial abilities were more useful and more highly regarded, partly for the sake of including a wide variety of tasks and partly for the sake of inflating male IQs to equal those of women.
Z.M. Davis: These days I generally try not to identify as a Transhumanist, Humanist, or even atheist except to people who I have reason to believe will react positively. I don't have the character to identify as Christian to those who will react positively to that, but maybe someday. Honestly, well, on Facebook I'm "Political Views", "Other", "Religion" "Philosopher and Bayesian". As the latter, any group identity that supposedly rests on belief is out, as I explicitly represent, when unpacked precisely, all my beliefs as probabilistic, even the normative ones. Ordinary discussion really doesn't consistently do this. The closest word, and it isn't close, is "Skeptic". There's no simple translation between "I believe X and I believe that my ignorance can be quantified" and anything thinkable to someone who doesn't believe that ignorance can be quantified. I really don't think that I'm engaged in moral intimidation towards Angel at all, but I don't think you are being petty. As I mentioned, I think that its a serious concern if people are and is worth bringing up if there is a reason to suspect it. There's a difference between saying "It seems to me that you are probably a pretty bad person so I am not going to trust you" and "you should stop thinking those thoughts because having those thoughts means that you are a bad person". Ironically, my guess is that with the right audience (probably a VERY small audience, I don't mean academic feminists in general) some talk about "privilege" and "dominance" would work well for clarifying this point, but with most audiences it would just add to any confusion.
frelkins: thanks for the references.
Minor note: The most appealing characters in The Odyssey are female, namely Athena and Penelope.
Laura: Men and women have the same average IQ and at a best guess the same 'g' (itself an astounding scientific finding given how different they are neuroanatomicaly, but well validated), but differences in specific abilities are so large that people with opposite gender typical relative verbal and visuo-spatial abilities are considered learning disabled (Unless they were Jewish, oddly, as massively superior verbal to visuo-spatial abilities are common among Jews and thus not considered abnormal). Is it terrible to belong to groups that have known strengths and weaknesses or for the weaknesses to be invoked as an explanation of an observed error? Why be upset by explaining one observation in terms of another established one unless arguments are soldiers? Since I'm Jewish I sometimes explain my visuo-spatial errors in the same group terms. Likewise, I imagine that you would not feel any shame about not being able to lift something that a man would be able to lift easily, and that in Africa, where women know how to carry things on their heads in a mechanically efficient manner that depends on broader hips, men probably don't feel shame at not being able to carry something that women can carry easily. I can tell you for sure that in Kazakhstan men aren't ashamed of lacking the basic skills to take care of themselves. So, are only members of a group ever allowed to bring attention to that group having any weaknesses at all, even if it is generally acknowledged that the group in question also has strengths? One problem with this approach to gender relations is that without the framework of group specific strengths people are liable to interpret failures as individual general weaknesses and to underestimate the general abilities of individual members of the opposite gender in the name of protecting the honor of the opposite gender. That seems worse to me.
Grant: I would second everything in your post except the last paragraph.
Angel: I sincerely do apologize for making you feel bad, and I certainly won't use your name in the future, as hurting people is bad, hurting people in ways that make them likely to behave worse is worse, and my not trusting the person I harm or not considering them to have good intentions doesn't make it better.
That said, NO, it is NOT wrongwrongwrong to tell people that its a bad idea to adopt such and such a label because the people who typically use that label are in some respect immoral, crazy, or even simply widely believed to be so. If you call yourself a Communist I will point out that you are choosing a label identified most strongly with mass murderers, and if you call yourself a Christian, one identified with warmongering anti-rationalists. Doing this is beneficial. If you call yourself a 'communist', you really will probably end up with an inappropriately positive attitude towards the Soviet Union even as you denounce it as "not really communism". If you call yourself a Muslim you really will probably end up with a level of sympathy for Islamic terrorists that you lack for terrorists of other types even as you insist that they are misguided and that "Islam is really a religion of peace".
Labels are in general destructive of rationality. In so far as they are useful, it is because their signaling value exceeds their emotional cost. It is totally appropriate to emphasize the negative messages that a label will send to someone who is considering adopting it, and the main source of such negative messages comes from making claims of similarity to all others who are using that label. In the case of feminism it is totally clear to me that many of the more prominent feminists endorse positions diametrically opposed to equal rights and to rational thought and it sure seemed to me like you did as well based on your posts.
At this point, I still don't believe with p>50% that you favor rational thought or equal rights, but I'm willing to make that assumption for argument's sake if you wish to stick around. I don't really prefer for you to stick around, but I do think that there is a very small chance of my learning something very important by your doing so, so I'll do what I can not do discourage your continuing presence here. Not in exchange, but rather as a token of your commitment to sincere deliberation I would like to ask you to internally and externally taboo "privilege" and "dominant" while you are here. These concepts seem to me to be too loaded and amorphous to contribute to the dialog.
Angel: I really don't think that in the modern world we need a label for "someone who believes in equal rights for men and women" any more than we need a label for "someone who thinks that we shouldn't have slavery" outside of the Muslim world. Abolitionist was once a mark of distinction, now it isn't. A more useful definition but extremely generous definition of feminism, one that would make me very tentatively feminist, would be something like "someone who believes that many/most cultural institutions and basic assumptions need to be critically reevaluated with attention to the fact that women have historically been denied adequate input into their creation". A hostile definition of feminism, but one that I think based on your posts seems to describe you, is a person who believes that the historical oppression of women provides a general-purpose excuse for condemning anyone a feminist wishes to condemn and for reaching any conclusion a feminist wishes to reach. It's a variant of the general problem of knowing about biases and using that knowledge to absolve you from the necessity of engaging with people's arguments, choosing instead to simply point out that they could be biased. Really, this range of possible interpretations of the word, which only scratches the surface of possibilities, further suggests that the word doesn't convey enough signal to justify the confusion it causes.
All: I'm really disappointed that no-one else seems to have found my "after the FAI does nothing" frame useful for making sense of this post. Is anyone interested in responding to that version? It seems so much more interesting and complete than the three versions E.C. Hopkins gave.
Dynamically: My "moral philosophy" if you insist on using that term (model of a recipe for generating a utility function considered desirable by certain optimizers in my brain would be a better term) is the main thing that HAS told me to steal, cheat, and murder. Simpler optimization patterns based on herd behavior, operant conditioning, moderately strong typical male primate aversions to violence, projections of parental authority through internalized neural agents etc have told me not to do those things and have won enough attention from the more complex optimizers to convince them (since the complex optimizers can reflect and be convinced of things) not to do so after all, and upon examination those simpler patterns have mostly turned out to be right judged by the standards of the moral philosophy. On a few occasions that I am aware of my conditioned etc morality was very wrong (judged reflectively), and possibly on a few other occasions, but they were much much less wrong than the occasions on which they were right and casual examination of my reflective self was in doubt.
The way I frame this question is "what if I executed my personal volition extrapolating FAI, it ran, created a pretty light show, and then did nothing, and I checked over the code many times with many people who also knew the theory and we all agreed that it should have worked, then tried again with completely different code many (maybe 100 or 1000 or millions) times, sometimes extrapolating somewhat different volitions with somewhat different dynamics and each time it produced the same pretty light show and then did nothing. Lets say I have spend a few thousand years on this while running as an upload. Now what?"
In this scenario there's no optimization reason I shouldn't just execute cached thoughts. In fact, that's pretty much what anything I do in this scenario amounts to doing. Executing cached thoughts does, of course, happen lawfully, so there is a reason to dress in black etc in that sense. I used to be pretty good at writing some sad but mostly non-gloomy poetry and denouncing people as fools. Might be even more fun to do that with other modified upload copies of myself. When that got old, maybe use my knowledge of FAI theory to build myself a philosophy of math oracle neural module. Hard to guess how my actions would differ once it was brought on-line. It seems to me that it might add up to normality because there might be an irreducible difference between utility for me and utility for an external AGI even if it was an extrapolation of my volition, but for now I'm a blind man speculating on the relative merits of Picasso and Van Gogh.
Honestly I'm much less concerned about this scenario than I once was. Pretty convinced that there are ways to extrapolate me that do something even if they discover infinite computing power.
Dynamically linked: No-one but nerds and children care what moral philosophies say anyway, at least, not in a way that effects their actions. You, TGGP and Unknown are very atypical. Poke is much closer to correct. If anything, when the dust settled the world would be more peaceful if most people understood the proof.
Eric Mesoy: If utilities = 0 then dying from malnourishment isn't horrible.
Andy M: Your answer sounds more appropriate for someone fairly shallow and 20 years old who discovers that the world or his life will end in 6 months than for someone for whom utilities are set to zero or morality is lost.
Constant Pablo and especially Sebastian: Clearly thought! I should probably start reading your comments more carefully in the future.
Laura: Why unsympathetic? My guess is that you still confuse my and Eliezer's aspirations with some puerile Nietzschean ambition. I like who I am now too thank you very much, and if my extrapolated volition does want to replace who I am it is for reasons that I would approve of if I knew them, e.g. what it will replace me with is not "completely different, incomprehensible, and unsympathetic". That's the difference between a positive and a negative singularity. Death isn't abhorrent, life/experience/growth/joy/flourishing/fulfillment, rather, is good, and a universe more full of them more good than one less full, whether viewed from inside or from outside. Math is full of both death and flourishing and is not lessened by the former.
Phil: Very entertaining and thoughtful post.
Laura: It seems to me that when you apply the feminist label you might want to think about the risk of being understood to mean "like Angel". If that prospect doesn't appall you you may want to read her posts more carefully. It's not just normal academic dishonesty. Rather, its a very distinctive establishment of moral asymmetries mixed with postmodernism that characterizes major strains of modern (second and third wave?) feminism. Maybe you want the term "first wave feminist" but that's too pretentious.
Bambi: The particular reason for blogging rather than rum is that the math says he blogs here and now. The future isn't immune to our actions, it is what it is as the result of our actions, which likewise are what they are. We cause it to be in the same manner that the earlier states of a Turing Machine cause the later states to be.
Eliezer: I really like this post, but it seems to me that empirically it was substantially a cultural practice in philosophy, including Kant etc, that enabled those early 20th century Germans (and only those people, in that particular culture, with that particular philosophical tradition) to seem, vaguely a significant subset of those assumptions that they did know existed but that other philosophers and lay people didn't know existed. That philosophy also lead them down some wrong roads, such as towards thinking mind was fundamental rather than emergent, and it certainly didn't enable them to see all of the assumptions that they didn't know existed, but there seems to be a known partial reason that the quantum revolution was so local, one credited by some of the physicists in question.
For what it's worth, I have only a lay understanding of quantum physics, still don't really know what you mean by configurations and amplitudes, and was able to see, fairly easily, the assumption that "Bob" in your assumption didn't see, basically about particles being "things" with "properties" attached to them, (an assumption that Chalmers, in "The Conscious Mind" seems to know is rejected by physics but to find it impossible to reject, leading him to mention his disturbance in a physical view of particles as "pure causal flux", which I would call "pure relationship" and which at least a few philosophers surely mean by "radical emergence") although I would have described it somewhat differently, e.g. not by explicit reference to technical information that I didn't have.
I don't think that the problem is that it is impossible with effort and training to learn to recognize one's blind spots a-priori. Rather, I think that philosophy attracts many kinds of people, only one of which is the type of person who has a talent that he wants to develop in recognizing his blind spots. Philosophy then provides, to different extents in different places and times, some training in this skill and some reward of status for the development of it. Currently, it seems to me that neither Analytic nor Continental philosophy provides significant training or status relating to this as opposed to other skills. More particularly, both seem to provide far less such training or reward in status than contemporary theoretical physics, theoretical computer science and probably some parts of math.
The main problem, it seems to me, relates to this issue of rewarding with status. In physics, ultimately status goes to those who make the correct predictions enabling correct beliefs to actually attain dominance in the field even if they are counter-intuitive (or too intuitive to qualify as 'deep'), while in philosophy, without experiments, correct beliefs always exist at a very low incidence at equilibrium, far less popular or 'official' than clever descriptions of those cognitive illusions such as empty labels http://lesswrong.com/lw/ns/empty_labels/ (in this case, the particle without the mathematical relationships it participates in) that act as attractors to human naive ontology. As a result, the average physicist is better at this type of philosophy than the average philosopher is, while the average highly esteemed physicist is astronomically better at it than the average highly esteemed philosopher.
BTW I'm not really convinced that "Bob" would be correct in "any classical universe", or even that classical universes are conceivable rather than apparently conceivable.
Poke: Are you sure about mineralogy and physics as foundations of modern geology?
Patrick:
I agree that something roughly along the lines of what you are discussing can be done and is unavoidable. I am primarily attempting to refute the proposal that it is or can be corrected to become Bayesian, and hence the proposal that the process that we use to do things like this stands with the same sort of logical foundations as Bayesian reasoning does. It definitely seems to me that strictly speaking, once you remove logical omniscience, unless you replace it with some very specific abstraction (most of which have their own problems) you need to assign probabilities to "The RH can be proved", "The RH can be disproved", "The RH is undecidable from ZFC", "The RH can be proved AND the RH can be disproved", "The RH can be proved AND the RH can be disproved AND The RH is undecidable from ZFC" "The RH can be disproved AND The RH is undecidable from ZFC" etc. In practice we can apportion zero probability to the latter cases, but only conditional upon the quality of our reflection being perfect, which we know to be false, and only after SOME reflection. It seems to me that as we assign probabilities we have to do reflection that moves our estimates continually.
I second tabooing probability, but I think that we need more than two words to replace it. Casually, I think that we need, at the least, 'quantum measure', 'calibrated confidence', and 'justified confidence'. Typically we have been in the habit of calling both "Bayesian", but they are very different. Actual humans can try to be better approximations of Bayesians, but we can't be very close. Since we can't be Bayesian, due to our lack of logical omniscience, we can't avoid making stupid bets and being Dutch Booked by smarter minds. It's therefore disingenuous to claim that vulnerability to Dutch Books is a decisive argument against a behavioral strategy. Calibrated confidence is the strategy that we can try to use to minimize our vulnerability to being Dutch Booked by people who aren't smarter than we are but who know exploits in our heuristics. They tend to be much much closer to 50% than Bayesian confidences, and are pretty much unavoidably subject to some framing based biases as a result.
To make the simulation really compelling it has to include some sort of assortative mating.
No Mike, your intuition for really large numbers is non-baffling, probably typical, but clearly wrong, as judged by another non-Utilitarian consequentialist (this item is clear even to egoists).
Personally I'd take the torture over the dust specks even if the number was just an ordinary incomprehensible number like say the number of biological humans who could live in artificial environments that could be built in one galaxy. (about 10^46th given a 100 year life span and a 300W (of terminal entropy dump into a 3K background from 300K, that's a large budget) energy budget for each of them). It's totally clear to me that a second of torture isn't a billion billion billion times worse than getting a dust speck in my eye, and that there are only about 1.5 billion seconds in a 50 year period. That leaves about a 10^10 : 1 preference for the torture.
The only considerations that dull my certainty here is that I'm not convinced that my utility function can even encompass these sorts of ordinary incomprehensible numbers, but it seems to me that there is at least a one-in-a-billion chance that it can.
I think that Robin's point solves this problem, but doesn't solve the more general problem of an AGI's reaction to low probability high utility possibilities and the attendant problems of non-convergence.
The guy with the button could threaten to make an extra-planar factory farm containing 3^^^^^3 pigs instead of killing 3^^^^3 humans. If utilities are additive, that would be worse.
Robin: Great point about states with many people having low correlations with what one random person can effect. This is fairly trivially provable.
Utilitarian: Equal priors due to complexity, equal posteriors due to lack of entanglement between claims and facts.
Wei Dai, Eliezer, Stephen, g: This is a great thread, but it's getting very long, so it seems likely to be lost to posterity in practice. Why don't the three of you read the paper Neel Krishnaswami referenced, have a chat, and post it on the blog, possibly edited, as a main post?
"The paper I referenced:
Vann McGee (1999)
An airtight Dutch book
Analysis 59 (264), 257â265.
Posted by: Neel Krishnaswami | October 20, 2007 at 06:29 PM"
Rob: I think that some psychologists might say something like the following. Confirmation bias causes new evidence to accumulate in favor of existing beliefs. Subsequent to the accumulation of such evidence, the refutation of the original evidence for a belief will not eliminate the belief. When the beliefs in question are normative evaluations, this is called the "halo effect".
Initial evidence that something, for instance ethical behavior, is good because it leads to Santa giving presents could lead to a bias in favor of noticing other evidence that ethical behavior is good, such as it eliminates some of the cognitive burden of lying, and it makes other people happy thus promoting vicarious pleasure. Sensitivity to these possibilities may lead to their better exploitation, and thus to a history of Pavlovian reinforcement from good behaviors.
That being said, much of Jewish and to a lesser degree Catholic law consists of the justification of reversion to preferred morality where it differs (usually in the sense of being much more ethical as modern seculars would understand the term) than the morality promoted by their theoretical authority figure, implying a (fortunate) failure of internalization of religious morality. More speculatively, some scholars have blamed the barbarities of the Soviets and NAZIs on (among other things) the reversion to a tribal/nationalist preferred morality which Christianity had advocated replacing, or simply to conformity or sadism in response to a moral vacuum. Unfortunately for this thesis, the Japanese, with a presumably more resilient internalized morality, committed comparable though somewhat lesser atrocities, suggesting, at best, that reverting to tribalism/nationalism is not much worse for outsiders than never having abandoned it. Unfortunately for this discussion, there are no obvious examples of maturely secular nations with universalist ethical axioms fighting major wars, but that fact could also be seen as tending to confirm the above speculation.
TGGP: I don't think that what you are describing would be considered by most people to constitute a disbelief in morality. Instead, I think it would be considered an atypically reflective and possibly slightly atypically targeted belief about what being means.
Rob Spear: "good old fashioned Pavlovian conditioning" requires that rewards and punishments (reinforcers) be associated very closely in time with the behaviors being reinforced. Santa doesn't do this. Neither does god. This is NOT a minor quibble, but rather is a critically important distinction. Evolutionary psychologists have discovered a few instances of non-old-fashioned Pseudo-Pavolvian conditioning, such as the development of aversions to foods as a result of sickness hours after eating (still not months or years after) and immediate acquisition of certain phobias (for instance, fear of snakes) without the need for repetition, but these are due to very distinctive context-specific mechanisms. There is no evidence that any analogous distinctive Pseudo-Pavolvian mechanism is at work in generating "Santa efficacy".
Rob: In practice there is a HUGE difference. If you behave morally when no0one is watching you, new information doesn't threaten your moral foundations, as your morality is grounded in your preferences. If you believe that you are always being watched then your moral behavior will be grounded in supposed facts about the world. In this case, evidence that undermines your belief in those facts undermines your morality, leading in the direction of Nietzsche's "total eclipse of all values" as the inevitable consequence of the "twilight of the idols".
There are also important practical consequences to following orders as opposed to acting from one's own initiative. Psychologically, the former will create resentment and dissatisfaction while the latter will not. The latter will favor initiative, and active pursuit of the best satisfaction of one's (hopefully ethical) desires, while the former will focus only on satisfying some perceived standard of acceptability, which implies, among other things, not seeking out better third alternatives when presented with a false dilemma and evidence that one flawed alternative is considered to be permissible. Actions "above the call of duty" depend entirely upon self-driven internalized ethics.
I think that the social sciences seem to still be following the Greek paradigm. Exceptions are excused and generally ignored rather than studied in more depth. New theories are rarely asked to explain the findings that supported old theories. Outliers are dropped, partially to make ignoring exceptions easier.
I would say that I referred to ontological types with an informational flavor rather than a "materialist" flavor, but since the process doing the referring is an informational process, what other possibility could have been addressed? Whereof one cannot speak, thereof one must be silent.
The point is not that a qualitatively more intelligent being could not design a universe to cancel a qualitatively less intelligent being's technology, but rather that an unintelligently (randomly subject to anthropic constraints, for instance) selected set of naturalistic laws could not plausibly generate such effects. You could always have a hands-on god individually deciding what happens in each situation (monadology?), but that's just a kid playing with dolls, not a universe. Also, "things" in modern physics are defined in terms of their relationships to one another, Chalmers' "pure causal flux". To change the relationships is to change the things, and to eliminate the relationships is to eliminate the things. For instance, objects have mass and mass a set of relationships to energy and to other massive objects mediated by space-time. You can put God in the mediative role of space-time without any impact, but as soon as God doesn't obey the same simple set of laws when mediating the effect of mass, in what sense do objects still have mass at all?
Funny, I always read "A Fire Upon The Deep" as a commentary upon this very point. It seemed to me that Vinge was rubbing people's faces in the fact that our intuitions about technology, where each discovery has an associated "importance" lisp token, doesn't correspond to the way the world works. The Zones correspond to something that exists in the minds of people thinking about the future, but doesn't correspond to anything that could possibly turn out to exist in a consilient monistic world (as opposed to the Platonistic internal world of stories and computer games). The story seemed to me to be an expose of the lack of depth implicit it non-singularitarian world models given the necessity of mind in monistic worlds emerging from a manipulable substrate.
OTOH, no-one else seems to read it that way, and "A Deepness In The Sky" was deeply disappointing by dropping the theme, so I guess that my read wasn't intended after all.
Well, at the very least women constitute half of society, it's certainly acceptance within that half. I actually think that it's actually acceptance more broadly though. Women are arguably not accepted my men in general, but in so far as they are accepted it is only in a few narrow domains, primarily science, engineering, and big business that women do best by adhering to men's norms. Actually though, emotional suppression is only normative among men in science, in the military, and in low status positions. Enthusiasm (irrational exuberance) is the ultimate business virtue. If one doesn't claim a level of confidence that can't possibly be justified one is simply not a contender for venture capital or angel investor money. In a hierarchy, one's not suitable for upper management or sales. Beyond that, almost all social elites are, in large measure, "emotional expression professionals". Actors and actresses are the most obvious example of this, but I would say that this is also true of athletes, artists, and other performers and entertainers, religious leaders, and politicians. Al Gore was dismissed with a characterization of "wooden". Hitler practiced his emotional expressions for hours in front of a mirror.
It seems to me that social consensus accepts expression of strong feelings by women, just not by men.
It might be worth considering whether metaethicists, theologians, and astrologers from radically different backgrounds tended to converge upon some core set of conclusions substantially different from those reached by non-metaethicists. Metaethical, astrological or theological equivalents of Darwin and Wallace or of Newton and Leibniz would be significant confirmation of their disciplinary soundness.
My causal account of moral directionality is that cognitive resources are scarce and that people try to conserve them. One class of cognitive resource people try to conserve is complexity of model, driving a tendency towards parsimonious explanations. In other words, ethical consistency is a sub-set of consistency in general, which is a luxury good. As non-cognitive wealth increases relative to cognitive wealth people purchase more parsimony and noise in early highly random but constrained ethical systems gets locked onto strong attractor dynamics. Islamists, whose meme complexes come from desert nomads with no parsimonious “do onto others” statement explicitly claiming to be the core of ethics, become more misogynistic and violent. Almost everyone else, all of whom hold meme-complexes originating in urban life in the axial age and possessing such a statement, expands their circle of empathy. We don’t know what happens to rich hunter-gatherer cultures, as we haven’t seen any.
Eli; what did you think as a young child in a religious environment?
It seems clear that there is a systematic direction in most or all cultures towards application and generalization of moral/ethical vocalizations as wealth increases. There is a less clear trend towards broadening circles of moral consideration, but this may be an instance of the first trend.
Gordon, I recommend the Satiricon of Petronius for some fantastic confirmation of your model of gambling applied to life success in general. It's fairly difficult for Americans, raised on meritocracy, and perhaps before than on the assumption of wise and just gods, to relate to the strong desire, expressed by many characters, for an unjust and capricious world. I suppose that if people imagine "lucky", "blessed", or "well fated" to be personal traits they can then easily believe that they are above average on those traits. The last, in particular, is not easily disconfirmed.