Open thread, September 2-8, 2013

As someone pointed out to me when mentioning this to them, to be a candidate for Great Filter there would need to be something intrinsic about how planets are formed that cause these two types of environments to be mutually exclusive, else it seems like there isn't sufficient reduction in probability of their availability. Is this actually the case? Perhaps user:CellBioGuy can elucidate.

(Are you Adele Lack Cotard?)

I'm also interested in hearing from you again about this project if you decide to not complete it. Rock on, negative data!

I read the sequences and a bunch of other great old main posts but now mostly read discussion. It feels like Main posts these days are either repetitive of what I've read before, simply wrong or not even wrong, or decision theory/math that's above my head. Discussion posts are more likely to be novel things I'm interested in reading.

Selection bias alert: asking people whether they have transitioned to reading mostly discussion and then to mostly just open threads in an open thread isn't likely to give you a good perspective on the entire population, if that is in fact what you were looking for.

I read the Sequences as they were posted; Main posts now rarely hold my interest the same way. Eliezer's writing is just better than most people's.

I predict that some people will have been through the sequences, which are Main posts, but then mainly cared about discussion. I suspect it has to do with Morning Newspaper Bias - the bias of thinking that new stuff is more relevant, when actually it is just pointless to read most of the time, only scrambles your mind, and loses value very quickly.

The lower the barrier to entry, the more the activity. Thus, more posts are on Discussion. My hypothesis is that this has worked well enough to make Discussion where stuff happens. c.f. how physics happens on arXiv these days, not in journals. (OTOH, it doesn't happen on viXra, whose barrier to entry may be too low.)

I've definitely noticed this in my use of LW. I find that the open threads/media threads with their consistent high-quality novelty in a wide range of subject areas are far more enjoyable than the more academic main threads. Decision theory is interesting, but it's going to be hard to hold my attention for a 3,000 word post when there are tasty 200-word bites of information over here.

I'll admit that much of the main sequence are too heavy to understand without prior knowledge, so I find discussions much easier to take in, and many times I end up reading a sequence because it was posted in a discussion comment. For me discussion posts are like the gateway to Main.

My experience is similar. I read the sequences as they were published on OB, then when the move over to LW happened I just subscribed to the RSS feed and only read Promoted posts for quite a few years. Only about a year ago I actually signed up for an account here and started posting and reading Discussion and the Open Thread.

For more on this topic, see for example these posts:

• Evolutionary Psychology
• Thou Art Godshatter
• Rebelling Within Nature
• The Gift We Give To Tomorrow

Anyone tried to use the outside view on our rationalist community?

I've seen a few attempts, mostly from outsiders. The danger involved there is an outsider has difficult picking the right reference class- you don't know how much they know about you, and how much they know about other things.

The things that the outside view has suggested we should be worried about that I remember (in rough order of frequency):

• Being a cult.

• Being youth-loaded.

• Optimizing for time-wasting over goal-achieving.

Here are two critiques I remember from insiders that seem to rely on outside view thinking: Yvain's Extreme Rationality: It's Not That Great, patrissimo's Self-Improvement or Shiny Distraction: Why Less Wrong is anti-Instrumental Rationality. Eliezer's old posts on Every Cause Wants To Be A Cult and Guardians of Ayn Rand also seem relevant. (Is there someone who keeps track of our current battle lines for cultishness? Do we have an air conditioner on, and are we optimizing it deliberately?)

One of the things that I find interesting is in response to patrissimo's comment in September 2010 that LW doesn't have enough instrumental rationality practice, Yvain proposed that we use subreddits, and the result was a "discussion" subreddit. Now in September 2013 it looks like there might finally be an instrumental rationality subreddit. That doesn't seem particularly agile. (This is perhaps an unfair comparison, as CFAR has been created in the intervening time and is a much more promising development in terms of boosting instrumental rationality, and there are far more meetups now than before, and so on.)

There's also been a handful of "here are other groups that we could try to emulate," and the primary one I remember was calcsam's series on Mormons (initial post here, use the navigate-by-author links to find the others). The first post will be particularly interesting for the "outside view" analysis because he specifically discusses the features of the LDS church and LW that he thinks puts them in the same reference class (for that series of posts, at least).

I'd like to here point out a reference class that includes if I understood things right: the original buddhism movement, the academic community of france around the revolution, and the ancient greek philosophy tradition. More examples and a name for this reference class would be welcome.

I include this mainly to counterbalance the bias towards automatically looking for the kind of cynical reference classes typically associated with and primed by the concept of the outside view.

Looking at the reference class examples I came up with, there seems to be a tendency towards having huge geniuses at the start that nobody later could compete with, wich lead after a few centuries to dissolving into spinoff religions dogmatically accepting or even perverting the original ideals.

DISCLAIMER: This is just one reference class and the conclusions reached by that reference class, it was reached/constructed by trying to compensate for a predicted bias with tends to lead to even worse bias on average.

I think the cult view is valuable to look at some issues.

When you have someone asking whether he should cut ties with his nonrational family members is valuable to keep in mind that's culty behavior.

Normal groups in our society don't encourage their members to cut family ties. Bad cults do those things. That doesn't mean that there's never a time where one should rationally advice someone to cut those ties, but one should be careful.

Given the outside view of how cults went to a place where people literally drunk kool aid I think it's important to encourage people to keep ties to people who aren't in the community.

Part of why Eliezer banned the basilisk might have been that having it more actively around would push LessWrong in the direction of being a cult.

There's already the promise for nearly eternal life in a FAI moderated heaven.

It always useful to investigate where cult like behaviors are useful and where they aren't.

Pardon me, but - what is the trust boostrapping involved?

Thank you. All I need is hand held spray thermos to make Australia a viable working vacation. I have a strong irrational aversion to spiders. This is much more acceptable than the home made flamer.

Does this also work with macaques, crows or some other animals that can be taught to use money, but didn't grow up in a society where this kind of money use is taken for granted?

I suspect that finding people on the borderline between the categories and giving them a nudge is part of the solution to this problem.

What do you need PCs to do that NPCs cannot do? Zeroing in on the exact quality needed may make the problem easier.

Take the leadership feat, and hope your GM is lazy enough to let you level them. More practically, is it a skills problem or as I would guess an agency problem? Can impress on them the importance of acting vs not? Lend them the Power of Accountability? 7 habits of highly effective people? Can you compliment them every time they show initiative? etc. I think the solution is too specific to individuals for general advice, nor do I know a general advice book beyond those in the same theme as those mentioned.

In loading trucks for warehouses, some OR guys I know ran into the opposite problem- they encoded all the rules as constraints, found a solution, and it was way worse than what people were actually doing. Turns out it was because the people actually loading the trucks didn't pay attention to whether or not the load was balanced on the truck, or so on (i.e. mathematically feasible was a harder set of constraints than implementable because the policy book was harder than the actuality).

(I also don't think it's quite fair to call the OR approach 'punting', since we do quite a bit of optimization using heuristics.)

The more I think about it the more I believe that a Friendly Singleton AI is the only way for us humans to survive.

So you want a god to watch over humanity -- without it we're doomed?

FAI mitigates other existential risks of nature, unknowns, human cooperation (Mutually Assured Destruction is too risky), as well as hostile intelligences; both human and self-modifying trans-humans. My credence – that without FAI, existential risks will destroy humanity within 1,000 years – is 99%.

I find it unlikely that you are well calibrated when you put your credence at 99% for a 1,000 year forecast.

Human culture changes over time. It's very difficult to predict how humans in the future will think about specific problems. We went in less than 100 years from criminalizing homosexual acts to lawful same sex marriage.

Could you imagine that everyone would adopt your morality in 200 or 300 hundred years? If so do you think that would prevent humanity from being doomed?

If you don't think so, I would suggest you to evaluate your own moral beliefs in detail.

It's evidence against them being a person whose opinion is strong evidence of B, which means it is evidence against B, but it's probably weak evidence, unless their endorsement of B is the main thing giving it high probability in your book.

I don't know if there's a name for this, but I definitely do it. I think it's perfectly legitimate in certain circumstances. For example, the more B is a subject of general dispute within the relevant grouping, and the more closely-linked belief in B is to belief in A, the more sound the heuristic. But it's not a short-cut to truth.

For example, suppose that you don't know anything about healing crystals, but are aware that their effectiveness is disputed. You might notice that many of the same people who (dis)believe in homeopathy also (dis)believe in healing crystals, that the beliefs are reasonably well-linked in terms of structure, and you might already know that homeopathy is bunk. Therefore it's legitimate to conclude that healing crystals are probably not a sound medical treatment - although you might revise this belief if you got more evidence. On the other hand, note that reversed stupidity is not truth - healing crystals being bunk doesn't indicate that conventional medicine works well.

The place where I find this heuristic most useful is politics, because the sides are well-defined - effectively, you have a binary choice between A and ~A, regardless of whether hypothetical alternative B would be better. If I stopped paying attention to current affairs, and just took the opposite position to Bob Crow on every matter of domestic political dispute, I don't think I'd go far wrong.

I don't know if there is a name for it, but there ought to be one, since this heuristic is so common: the reliability prior of an argument is the reliability of the arguer. For example, one reason I am not a firm believer in the UFAI doomsday scenarios is Eliezer's love affair with MWI.

Yes, but in many cases it's very weak evidence. Overweighing it leads to the “reversed stupidity” failure mode.

Bayes' theorem to the rescue! Consider a crank C, who endorses idea A. Then the probability of A being true, given that C endorses it equals the probability of C endorsing A, given that A is true times the probability that A is true over the probability that C endorses A.

In equations: P(A being true | C endorsing A) = P(C endorsing A | A being true)*P(A being true)/P(C endorsing A).

Since C is known to be a crank, our probability for C endorsing A given that A is true is rather low (cranks have an aversion to truth), while our probability for C endorsing A in general is rather high (i.e. compared to a more sane person). So you are justified in being more skeptical of A, given that C endorses A.

It's a logical fallacy, but is something humans evolved to do (or didn't evolve not to do), so may in fact be useful when dealing with humans you know in your group.

Somewhat related: The Correct Contrarian Cluster.

ad hominem

Not that there's anything wrong with that.

The usual argument (which I agree with) is that "Death events have a negative utility". Once a human already exists, it's bad for them to stop existing.

Assuming Rawls's veil of ignorance, I would prefer to be randomly born in a world where a trillion people lead billion-year lifespans than one in which a quadrillion people lead million-year lifespans.

Existing people take priority over theoretical people. Infinitely so. This should be obvious, as the reverse conclusion ends up with utter absurdities of the "Every sperm is sacred" variety.

Mad grin

Once a child is born, it has as much claim on our consideration as every other person in our light cone, but there is no obligation to have children. Not any specific child, nor any at all. Reject this axiom and you might as well commit suicide over the guilt of the billions of potentials children you could have that are never going to be born. Right now.

Even if you stay pregnant till you die/never masturbate, this would effectively not help at all - each conception moves one potential from the space of "could be" to to the space of "is", but at the same time eliminates at least several hundred million other potential children from the possibility space - that is just how human reproduction works.

TL:DR; yes, yes they are. It is a silly question.

If by “old humans” you mean healthy adults, yes. If you mean this, no. (IMO -- YMMV.)

Death isn't just a negative for the dead person - it also causes paperwork and expenses, destruction of relationships, and grief among the living.

Yes.

Since PB users' calibrations are not yet good enough to see the future, you can easily avoid MoR spoilers by subscribing to the email or RSS alerts for new chapters & reading them as appropriate.

I used feed43 to create an rss feed out of recent predictions. Then I used feedrinse to filter out references to hpmor resulting in a safe feed. (Update: chaining unreliable services makes something even less reliable.)

You could do the same for the pages of recently judged or future or users you follow. I think feedrinse offers to merge feeds (into a "channel") before or after doing the filtering. But if you find someone new and just want to click on the username, you'll leave the safe zone. Even if you see someone you have processed, the username will take you to the unsafe page.

A better solution would be to write a greasemonkey script that modified each predictionbook page as you look at it.

The final feedrinse feed works in a couple of my browsers, but not chrome. Probably sending it through feedburner would fix it.

feed43 was finicky. The item search pattern was:

The regexp I used in feedrinse was /hp.?mor/
It is case insensitive and manages to eliminate "HP MoR:", "[HPMOR]", etc. It won't work if they spell it out, or just predict "Harry is orange" without indicating which story they're predicting about. In that case, someone will probably leave a hpmor comment, but this doesn't see such comments.

Is there a good way to avoid HPMOR spoilers on prediction book?

If you are skilled in the art of Ruby, then yes. Otherwise, maybe. People (myself included) have been complaining about the lack of tagging/sorting system on PB for quite some time, but so far, no one has played the hero.

To be honest, you sound a bit like a person who made a billion dollars and now tries to crowd-source a way to make ten billions. :-)

This book pbhyq uryc jvgu gur fgnzvan. Vg jbexrq sbe zl uhfonaq, jura ur gevrq vg n srj lrnef ntb.

Slow Sex seems to help move at least some people to move from good to great.

Practice makes perfect. I think a lot of good sex is intuitively reading your partner's signals and ramping things up/down with good timing in response to them. I think this is something you might be able to learn via logos but I think it's much more likely to be something you need to experience before you can get good at it. When to pull hair, when to thrust deeper, etc.

In general I and whoever I'm with have had more fun when I felt I had a good idea of what they wanted in the moment, which I think I've gotten better at mainly through practice.

Technically yes, but obviously if this is part of your motivation for doing so then thats a meaningful difference unless your parents also understood TDT and had that as part of their reason, so if in fact they did not (which they probably didn't since it wasn't invented back then) then this answer is entirely useless.

I've just looked and I have no idea either. If anyone wants to help there's a copy of the book here.

EDIT: The numbers in that copy are off by 1 from the book. "18.10" = "18-9" and so on.

A not-quite-rigorous explanation of the thing in 18.15:

E_aa is, by construction, only relevant to A. A_p was defined (in 18.1) to screen off all previous knowledge about A. So in fact, if we are given evidence E_aa but then given evidence A_p, then E_aa becomes completely irrelevant: it's no longer telling us anything about A, but it never told us anything about anything else. Therefore P(F|A_p E_aa) can be simplified to P(F|A_p).

Yeah, I don't see how those make sense either.

The term is also used more generally, to represent the idea that a new theory should reproduce the results of older well-established theories in those domains where the old theories work.

Sounds good to me, although that's not what I would have guessed from a name like 'correspondence principle'.

I suppose some minor difference is that this "law" is also applicable to meta-ethics, not just to physics. It's probably worth adding a link to the standard terminology to the LW wiki page.

This is the same basic idea as Benja's Parametric Polymorphism, with N in the post corresponding to kappa from parametric polymorphism.

The "superstition" is:

We make our compiler superstitious as follows: we pick an unused no-op of the target machine that should never occur in normal generated code and add an axiom that says that running this no-op actually bumps a counter that exists in the ether, and, if this unseen counter exceeds N, then it halts execution of the machine. Since we could have started with an arbitrary N, we will never reach a contradiction if we take the bottom out of the level descent, so long as we don't let the logic know we took the bottom out.

And from the section in Tiling Agents about parametric polymorphism (recommended if you want to learn about parametric polymorphism):

It is difficult to see what "believing in" $T_\kappa$ could correspond to in terms of the epistemic state of a rational agent. We believe from outside the system that $\kappa$'s intended interpretation is "any natural number," but this realization is apparently forbidden to the agent to whom $\kappa$ refers to some specific finite number about which nothing is known except that it plays a vital role in the agent's goal system. This seems like an odd mental state for a rational agent.

Anyway, it's interesting that someone else has a very similar idea for this kind of problem. But as mentioned in Tiling Agents, the "superstitious belief" seems like a bad idea for an epistemically rational agent.

So, in other words, absolutely no engagement with the actual ideas/arguments of the people the 'letter' is addressed to.

See http://en.wikipedia.org/wiki/Yvain and hover your mouse over the IPA pronunciation key.

Talk about it with other people. Ask a good friend to sit down with you and listen to you talking about the issue.

Wunderlist 2 has android (it only speaks english in the phone app, but it does portuguese in the normal online version.

it puts your tasks in the cloud so you can catch up with what you wrote in other services.

I'm amazed by David Allen's GTD at the moment, so I want to recommend it, despite still being on honeymoon effect.

I want to tack onto this and ask for a solution that provides some privacy, that is where I can run my own server.

I was on Astrid too. I switched to Wunderlist mostly because their import from Astrid worked correctly. Wunderlist is OK, though I can't say I'm completely satisfied with it. Its UI is laggy (on a Nexus 4!) and unreliable, for example the auto-sync often destroys the last task I just typed in, or when I accidentally tap outside the task entry box the text I just typed is lost forever.

I'm looking at alternatives, and the one I like the most so far is Remember the Milk. Last time I tried it (probably a year ago) it was rubbish, but the latest version has a clean and fast native Android GUI and some nice extra functionality (e.g. geofencing). I'm thinking about switching, but it doesn't have import from Wunderlist, so I'll have to move about 200 tasks manually.

I've been happily using http://www.rememberthemilk.com/ to manage my GTD system. It's got a simple, intuitive interface, both on desktop and on Android. I'm not sure if it has the reminder features you're after, since that's not something I've ever wanted.

What don't you understand? The 2 homo economicuses are aware that 'existence is suffering' especially when they are the butt of the humor, and rationally commit suicide.

http://tvtropes.org/pmwiki/pmwiki.php/Main/AntiHumor

I used to like liferea, but I don't have an up to date opinion as I switched to non-desktop RSS reading options.

It looks fine on Safari for the iPhone.

Site layout hasn't changed for me. Chrome on windows and safari on iphone.

since raised animals are vastly outnumbered by wild animals

That doesn't sound true if you weight by intelligence (which I think you should since intelligent animals are more morally significant). Surely the world's livestock outnumber all the other large mammals.

There's not a whole lot we can do now, so one thing I've heard suggested is to spread vegetarianism so that people will be more sympathetic to animals in general, and when we have the ability to engineer some retrovirus to make them suffer less or something like that, we'll care more about helping animals than not playing god.

Another possibility: nuke the rainforests.

This could be a Lovecraft horror story: "The Existential Diary of JMiller."

Week 3: These books are maddeningly incomprehensible. Dare I believe that it all really is just nonsense?

Week 8: Terrified. Today I "saw" it - the essence of angst - and yet at the same time I didn't see it, and grasping that contradiction is itself the act of seeing it! What will become of my mind?

Week 12: The nothingness! The nothingness! It "is" everywhere in its not-ness. I can not bear it - oh no, "not", the nothingness is even constitutive of my own reaction to it - aieee -

(Here the manuscript breaks off. JMiller is currently confined in the maximum security wing of the Asylum for the Existentially Inane.)

1. If you do not have a preexisting tendency for depression as a result of taking ideas seriously, you probably have nothing to worry about. If you are already a reductionist materialist, you also probably have nothing to worry about. Millions of college students have taken courses in existentialism. Almost all of them are perfectly fine. Even if they're probably pouring coffee right now.

2. In LW terms, it may be useful to brush up on your metaethics, as such problems are usually most troublesome about these kinds of ideas in my social circle. Joy in the Merely Real may also be useful. I have no idea how your instructors will react if you cache these answers and then offer them up in class, though. I would suggest not doing that very often.

3. In the event that the material does overwhelm you beyond your ability to cope, or prevents you from functioning, counseling services/departments on college campuses are experienced in dealing with philosophy-related depression, anxiety, etc. The use of the school counseling services should be cheap/free with payment of tuition. I strongly suggest that you make use of them if you need them. More generally, talking about the ideas you are learning about with a study group, roommate, etc. will be helpful.

4. Eat properly. Sleep properly. Exercise. Keep up with your studying. Think about things that aren't philosophy every once in a while. Your mind will get stretched. Just take care of it properly to keep it supple and elastic. (That was a really weird metaphor.)

When reading Merleau-Ponty it might help to also read the work of contemporary phenomenologists whose work is much more rooted in cognitive science and neuroscience. A decent example is Shaun Gallagher's book How the Body Shapes the Mind, or perhaps his introductory book on naturalistic phenomenology, which I haven't read. Gallagher has a more or less Merleau-Pontyesque view on a lot of stuff, but explicitly connects it to the naturalistic program and expresses things in a much clearer manner. It might help you read Merleau-Ponty sympathetically.

All of those weird books were written by humans.
Those humans were a lot like other humans.
They had noses and butts and toes.
They ate food and they breathed air.
They could add numbers and spell words.
They knew how to have conversations and how to use money.
They had girlfriends or boyfriends or both.
Why did they write such weird books?
Was it because they saw other humans kill each other in wars?
Was it because writing weird books can get you a lot of attention and money?
Was it because they remembered feeling weird about their moms and dads?
People talk a lot about that.
Why do they talk a lot about that?

Ignorance isn't bliss. If the course brings you in contact with a few Ugh fields that you hold that should be a good.

I think existentialism is very compatible w/ naturalism/reductionism. Existentialists just use a weird vocabulary. But one of the main points, I think, is coping with an absent/insane deity.

I suspect that warning was intended as a joke.

I would have a preference for never seeing such notifications.

For what purpose? Can't one simply comment or private message if they feel its necessary?

What happened with the Sequence Reruns? I was getting a lot out of them. Were they halted due to lack of a party willing to continue posting them, or was a decision made to end them?

I'm pretty sure they halted because they had gone through the Sequences. Final Words, the last rerun post, was published after Go Forth and Create The Art!, which is listed as the last of the Craft and Community sequence, which was the last of the Major Sequences.

What happened with the Sequence Reruns? I was getting a lot out of them. Were they halted due to lack of a party willing to continue posting them, or was a decision made to end them?

I never heard of such a decision and if it was made then it can be ignored because it was a bad decision. (Until power is applied to hinder implementation.)

If you value the sequence reruns then by all means start making the posts!

I personally get my 'sequence reruns' via the audio versions. Originally I used text to speech but now many of them have castify.

You cannot fail a personality test unless the person administering the test wants to filter out specific personality types that are similar to yours, for a process unrelated to the test itself (e.g. employment).

The thing is, most possible personalities seem to be considered undesirable by employers, and so many people simply resort to lying on these tests to present a favourable image to employers (basically: extrovert, conformist, "positive"/upbeat/optimistic, ambitious, responsible etc.). Looks like employers know about this, but don't care anyway, because they think that if you aren't willing to mold yourself into somebody else for the sake of the job, then you don't want the job enough and there are many others who do.

(Disclaimer: I'm an outsider to the employment process and might not know what I'm talking about. My impressions are gathered from job interview advice and job descriptions.)

You might be interested in the [Big Five] model of personality, which seems to be a rough scientific consensus, and is better empirically supported than other models. In particular, measures of conscientiousness have a relatively strong predictive value for things like grades, unemployment, crime, income, etc. Myers-Briggs-style tests that sort people into buckets ("You're an extrovert! You're an introvert!") are more common but don't seem to have a much predictive value except insofar as they reduce to something like the Big Five model.

However, from what I remember of applying to crappy jobs, you may not have taken a real test. I remember normal sounding items like, "I prefer large groups to small groups", mixed in with "trick" questions like, "If I saw my best friend stealing from the cash register, I would report him/her". I assume you got those right. Either way, you're expected to just lie and say you'd be the perfect, most hard-working employee ever and that cleaning toilets/selling shoes/washing cars is what you've dreamed of doing since you were five.

I've heard (second-hand, but the original source was a counselor for job-finding) that a trick for passing those, if it's a test that offers options from "Strongly disagree" to "Strongly agree", is to always pick one of the polarized ends ("Strongly" either). The idea seems to be that they'll prefer candidates who are less washy, have stronger convictions, etc.

Raqvat inzcver zheqref VF gernqvat yvtugyl: Vg'f n cbyvgvpny zbir. Gur orfg jnl gb trg uhznavgl abg gb ungr naq srne lbh vf gb or noyr gb pbasvqragyl gryy gurz gung gurl unir ab ernfba gb, naq gung lbh jnag bayl jung'f orfg sbe gurz. Zhpu nf Nzrevpn vf zvfgehfgrq va gur zvqqyr rnfg orpnhfr jr obzo jvgu unaq naq qvfgevohgr sbbq fhccyvrf jvgu gur bgure, crbcyr naq tbireazragf jvyy or zhpu yrff jvyyvat gb gehfg inzcverf vs gurl'er fgvyy xvyyvat crbcyr ng jvyy. Gur uhzna cbyvgvpny cbjref naq nyy gur uhznaf va gur jbeyq ner ahzrebhf rabhtu gung vs gur znfxrenqr oernxf, gur inzcverf jbhyq or va ZNWBE gebhoyr. Gur ovttrfg rkgnag guerng gb gur znfdhrenqr vf uhznaf orvat xvyyrq ol inzcverf. Fb fgbccvat xvyyvat uhznaf vf gur arkg fnsr fgrc jurgure lbh ner gelvat gb erirny inzcverf be pbaprny gurz.

Do you mean the end of Luminosity or the end of Radience?

The characters in those fics are also vastly more intelligent and conscientious than average. True, current school environments are stifling for gifted kids, but then they are also a very small minority. Self-directed learning is counterproductive for not-so-bright, and attempts to reform schools to encourage "creativity" and away from the nasty test-based system tend to just be smoke-screens for any number of political and ideological goals. Like the drunk man and the lamppost, statistics and science are used for support rather than illumination, and the kids are the ones who suffer.

There are massive structural problems wracking the educational system but I wouldn't take the provincial perspectives of HPMoR or related fiction as good advice for the changes with the biggest marginal benefit.

What concrete criteria would we like to see satisfied?

I think that's a bad question. I don't think that every school should follow the same criteria. It's perfectly okay if different school teach different things.

http://www.kipp.org/ would be an educational project financed by Bill Gates which tries to use a lot of testing. On the other hand you have unschooling and enviroments like Sudbury Valley School. I don't think that every child has to learn the same way. Both ways are viable.

When it comes to the more narrow rationality community I think there more thought about building solutions that educate adults than about educating children. If however something like Anki helps adults learn, there no real reason why the same idea can't help children as well.

Similar things go for the Credence game and predictionbook. If those tools can help adults to become more calibrated they probably can also help kids even if some modifications might be needed.

Without having the money to start a completly new school I think it's good to focus on building tool that build a particular skill.

If everyone agrees about how power is distributed fighting is unnecessary.

Fighting can be necessary when another person claims to have power that they actually don't have.

When is fighting good? When does fighting lead you to Win TDT style (which instances of input should trigger the fighting instinct and payoff well?)

Or even just CDT style. Human interaction is approximately an iterated prisoners dilemma without a fixed duration. Reputation concerns are sufficient to account for most of the (perceived and actual) benefit among humans. Then more can be attributed to ethical inhibitions on the 'pride' ethic.

Fighting makes a lot more sense in a tribe or in small groups/individuals of humans than it does now. A big argument with someone now will very rarely keep you from starving and will probably never get you a child. On the other, showing dominance in a situation where the women around you are choosing a mate out of 5 guys, will get you a lot more laid.

On LW, 'singularity' does not refer to a mathematical singularity, and does not involve or require physical infinities of any kind. See Yudkowsky's post on the three major meanings of the term singularity. This may resolve your physicist friend's disagreement. In any case, it is good to be clear about what exactly is meant.

Lack of cheap energy.

Ecological disruption.

Diminishing returns of computation.

Diminishing returns of engineering.

Inability to precisely manipulate matter below certain size thresholds.

All sorts of 'boring' engineering issues by which things that get more and more complicated get harder and harder faster than their benefits increase.

So my question is: What are the 'obvious' candidates for limits that take over before the all optimizable is optimized by runaway technology?

There aren't any that I'm aware of, except for "a disaster happens and everyone dies," but that's bad luck, not a hard limit. I would respond with something along the lines of "exponential growth can't continue forever, but where it levels out has huge implications for what life will look like, and it seems likely it will level out far above our current level, rather than just above our current level."

One calculation per planck time per cubic planck length in the future light cone.

It looks to me like you want a cryptographically secure pseudo-random number generator restricted to the output space {0, 1} and with a known seed. That's unbiased and intractable pretty much by definition, indexable up to some usually very large periodicity, and typically verifiable and simple to refer to because that's standard practice in the security world.

There's plenty of PRNGs out there, and you can simply truncate or mod their outputs to give you the binary output you want; Fortuna) looks like a strong candidate to me.

(I was going to suggest the Mersenne twister, which I've actually implemented before, but on further examination it doesn't look cryptographically strong.)

My first idea is to use something based on cryptography. For example, using the parity of the pre-image of a particular output from a hash function.

That is, the parity of x in this equation:

f(x) = n, where n is your index variable and f is some hash function assumed to be hard to invert.

This does require assuming that the hash function is actually hard, but that both seems reasonable and is at least something that actual humans can't provide a counter example for. It's also relatively very fast to go from x to n, so this scheme is easy to verify.

No candidates, but I'd like to point out that your unbiased requirement may perhaps be omitted, conditional on the implementation.

If you have a biased logical coin, you poll the coin twice until the results differ, and then you pick the last result when they do differ. That results in an unbiased logical coin.

My first instinct is to bet on properties of random graphs, but that's not my field.

No open thread for Jan 2014

Now there is. The next time you miss an open thread you can make one. A lot more people will see your comment than if you post in an old thread, and you might get a point or two of karma.

But would it be possible for an AI to have a "knowledge-about-problem-solving foom" instead, where the AI increases its intelligence not by modifying itself, but by increasing the amount of knowledge it has about how to solve problems?

My immediate reaction is, 'Possibly -- wait, how is that different? I imagine the AI would write subroutines or separate programs that it thinks will do a better job than its old processes. Where do we draw the line between that and self-modification or -replacement?'

If we just try to create protected code that it can't change, the AI can remove or subvert those protections (or get us to change them!) if and when it acquires enough effectiveness.

Why should an AI have to self-modify in order to be super-intelligent?

I'm not sure where the phrase "have to" is coming from. I don't think the expectation that we will build a self-modifying intelligence that becomes a superintelligence is because that seems like the best way to do it but because it's the easiest way to do it, and thus the one likely to be taken first.

In broad terms, the Strong AI project is expected to look like "humans build dumb computers, humans and dumb computers build smart computers, smart computers build really smart computers." Once you have smart computers that can build really smart computers, it looks like they will (in the sense that at least one institution with smart computers will let them, and then we have a really smart computer on our hands), and it seems likely that the modifications will occur at a level that humans are not able to manage effectively (so it really will be just smart computers making the really smart computers).

But doesn't making the FAI self-modifying make the problem much more difficult, since how we have to figure out how to make goals stable under self-modification, which is also a very difficult problem?

Yes. This is why MIRI is interested in goal stability under self-modification.

Commonly, humans have an amount of empathy that means that when they know about suffering of entities within their circle of interest, they also suffer. EG, I can feel sad because my friend is sad. Some people have really vast circles, and feel sad when they think about animals suffering.

Do you understand suffering yourself? If so, presumably when you suffer you act to reduce it, by not holding your hand in a fire or whatnot? Working to end suffering of others can end your own empathic suffering.

LWers seem to be pretty concerned about reducing suffering by vegetarianism, charity, utilitarianism etc.

With respect to vegetarianism there are a couple of vocal advocates. Don't assume this applies to a majority. "Utilitarianism" is also considered outright ridiculous by many (in contrast to the principles of consequentialism, expected value maximising and altruistic values, which are generally popular.)

Wow. I was so heavily downvoted while not even one of my arguments was refuted. Really?

Since nobody has any reason to reduce suffering other than 'I want to' / 'I feel so', I think I may conclude that utilitarianism is a great hobby for oneself but it is kind of hypocritical to say that "utilitarianism is for greater good" or something like this.

Therefore when you coerce other people or kill one to save three, you do this not because of "greater good" but because you like to coerce and kill.

Upd: I hope that guys who downvote this comment do have that reason and maybe they would even be so kind and share it with me.

"Killing people and taking their stuff" has a positive QALY per dollar.

Dubious. Did you factor in the resources wasted on police investigation, mourning etc? :-)

Why do you say that? Supposing diminishing marginal returns for addition resources, I don't see how you're going to get around the QALY loss from killing the person.

Not after taking into account practical considerations.

"Killing people and taking their stuff" has a positive QALY per dollar. GiveWell should check it out.

The second sentence actually doesn't follow from the first. Givewell investigating it has a negative expected value even if actually doing it (well) has positive value. Among other things it makes it harder for Robin Hoods to not get caught.

I can't remember the article where this was stated, but we have instincts for morality because following them made our ancestors more successful. They're their for our benefit, not each others'. It seemed to your ancestors that killing someone and taking their stuff would be a net benefit, and if they didn't have a built-in aversion they'd do it, and they would likely get caught and punished.

Do you have a specific institution in mind which engages in that practice?

Joking or not, this is not the sort of conversation we should be having.

Your arguments conflict with what is called the "orthogonality thesis":

Leaving aside some minor constraints, it possible for any ultimate goal to be compatible with any level of intelligence. That is to say, intelligence and ultimate goals form orthogonal dimensions along which any possible agent (artificial or natural) may vary.

You'll be able to find much discussion about this on the web; it's something that LessWrong has thought a lot about. The defender's of the orthogonality thesis would have issue with much of your post, but particularly this bit:

Why would an A.I. with no initial goal choose altruism? Quite simply, it would realize that it was created by other sentient beings, and that those sentient beings have purposes and goals while it does not. Therefore, as it was created with the desire of these sentient beings to be useful to their goals, why not take upon itself the goals of other sentient beings?

The question isn't "why not?" but rather "why?". If it hasn't been programmed to, then there's no reason at all why the AI would choose human morality rather than an arbitrary utility function.

An AI has to be programmed. For something like this: "Quite simply, it would realize that it was created by other sentient beings, and that those sentient beings have purposes and goals while it does not." to happen, you have to program that behavior in somehow, which already involves putting in the value of respecting one's creator, and respecting the goals of other sentient beings, etc... The same goes for the 'Utilitarian' and 'Egoist' AI's - these behaviors have to be programmed in somehow.

As the superintelligence is probably going to be beyond our abilities to fathom, there is a high degree of uncertainty, which suggests a uniform distribution. The probabilities therefore are 1/3 for each of altruism, utilitarianism, and egoism.

Why not split the egoism into a million different cases based on each specific goal? You can't just arbitrarily pick three possibilities, and then use a uniform prior on these. Because we know these different behaviors have to be programmed in, we have a better prior: we can use Solomonoff Induction. We also have to look at the relative sizes of each class - obviously there are many more AI designs that fall under 'Egoist' than your other labels. Combining this with Solomonoff Induction leads to the conclusion that the vast majority of AI designs will be unfriendly.

An AI Existential Crisis is also an extremely specific and complex thing for an AI design, and is thus extremely unlikely to happen - it is not the default, as you claim. This also follows by Solomonoff Induction. You are anthropomorphizing AI's far too much.

Your suggestion will almost certainly lead to an Unfriendly AI, and it will just plain Not Care about us at all, inevitably leading to the destruction of everything we value.

As the superintelligence is probably going to be beyond our abilities to fathom, there is a high degree of uncertainty, which suggests a uniform distribution. The probabilities therefore are 1/3 for each of altruism, utilitarianism, and egoism.

This is a very bad use of uniformity. Doing so with large categories is not a good idea, because someone else can come along and split up the categories in a different way and get a different distribution. Going with a uniform distribution out of ignorance is a serious problem.

You're demonstrating a whole bunch of misconceptions Eliezer has covered in the sequences. In particular, you're talking about the AI using fuzzy high level human concepts like "morals" and "philosophies" instead of as algorithms and code.

I suggest you try to write code that "figures out a worthwhile moral goal" (without pre-supposing a goal). To me that sounds as absurd as writing a program that writes the entirety of its own code: you're going to run into a bit of a bootstrapping problem. The result is not the best program ever, it's no program at all.