The Level Above Mine

In a few years, you will be as embarrassed of these posts as you are today of your former claims of being an Algernon, or that a logical paradox would make an AI go gaga, the tMoL argumentation you mentioned the last days, the Workarounds for the Laws of Physics, Love and Life Just Before the Singularity and so on and so forth. Ask yourself: Will I have to delete this, too ?

And the person who told you to go to college was probably well-meaning, and not too far from the truth. Was it Ben Goertzel ?

Despite all fallibility of memory, I would be shocked to learn that I had ever claimed that a logical paradox would make an AI go gaga. Where are you getting this from?

Ben's never said anything like that to me. The comment about going to college was from an earnest ordinary person, not acquainted with me. And no, I didn't snap at them, or laugh out loud; it was well-intentioned advice. Going to college is a big choice for a lot of people, and this was someone who met me, and saw that I was smart, and thought that I seemed to have the potential to go to college.

Which is to imply that if there's a level above Jaynes, it may be that I won't understand it until I reach Jaynes's level - to me it will all just look like "going to college". If I recall my timeline correctly, I didn't comprehend Jaynes's level until I had achieved the level of thinking naturalistically; before that time, to achieve a reductionist view of intelligence was my whole aspiration.

Although I've never communicated with you in any form, and hence don't know what it's like for you to answer a question of mine, or correct a misconception (you have, but gradually), or outright refute a strongly held belief...or dissolve a Wrong Question...

...You're still definitely the person who strikes me as inhumanly genius - above all else.

Unfortunately for my peace of mind and ego, people who say to me "You're the brightest person I know" are noticeably more common than people who say to me "You're the brightest person I know, and I know John Conway". Maybe someday I'll hit that level. Maybe not.

Until then... I do thank you, because when people tell me that sort of thing, it gives me the courage to keep going and keep trying to reach that higher level.

Seriously, that's how it feels.

So how does it work, in your opinion? Because “I may not be as brilliant as Jaynes or Conway, but that doesn't mean I can't do important things in my chosen field,” sounds suspiciously similar to how Hamming asserts that it works in “You and Your Research.” I guess you have a different belief about how doing important things in your chosen field works, but I don't see that you've explained that belief here or anywhere else that I've seen.

I don't suppose Marcello is related to Nadja and Josh Herreshoff?

I don't know if it helps, but while I've appreciated the things I've learned from you, my limited interaction with you hasn't made me think you're the brightest person I know. I think of you as more or less at my level — maybe a couple of standard deviations above or below, I can’t really tell. Certainly you're sharp enough that I'd enjoy hanging out with you. (Let me know the next time you're in Argentina.)

P.S. the impugnment of your notability has now been removed from your Wikipedia page, apparently as a result of people citing you in their papers.

Wait wait wait wait. Eliezer...are you saying that you DON'T know everything????

~runs off and weeps in a corner in a fetal position~

CatAI (1998): "Precautions"/"The Prime Directive of AI"/"Inconsistency problem".

My memory may fail me, and the relevant archives don't go back that far, but I recall Ben (and/or possibly other people) suggesting you going to college, or at least enroll for a grad program in AI, on the Extropy chat list around 1999/2000. I think these suggestions were related to, but not solely based on, your financial situation at that time (which ultimately led to the creation of the SIAI, so maybe we should be glad it turned out the way it did, even if, in my opinion, following the advice would have been beneficial to you and your work.)

I was curious how you'd react when you eventually realized you weren't as bright as you thought you were. The journey to full comprehension isn't complete yet, but it's interesting seeing this little bit unfold. For all your disdain of modesty arguments, your life makes for a great demonstration of how one can go wrong if they go unheeded.

I definitely see the "levels" phenomenon very often. Most people I meet who see me play a musical instrument (or 5 or 10 different ones) think I must be a genius at music - unless they're a musician, then they recognize me as an amateur with enough money to buy interesting instruments and enough skill to get a basic proficiency at them quickly.

And even with standard measures of intellect like rationality or math... I don't know that many of my friends who have read any of this blog would recognize you as being smarter than me, despite the fact that you're enough levels above me that my opinion of you is pretty much what "Not You" said above.

I can keep up with most of your posts, but to be able to keep up with a good teacher, and to be that good teacher, is a gap of at least a few levels. But aspiring to your level (though I may not reach it) has probably been the biggest motivator for me to practice the art. I certainly won't be the one who zips by you, but you've at least pulled me up to a level where I might be able to guide one who will down a useful path.

Up to now there never seemed to be a reason to say this, but now that there is:

Eliezer Yudkowsky, afaict you're the most intelligent person I know. I don't know John Conway.

I was curious how you'd react when you eventually realized you weren't as bright as you thought you were.

Your faith in math is misplaced. The sort of math smarts you are obsessed with just isn't that correlated with intellectual accomplishment. For accomplishment outside of math, you must sacrifice time that could be spent honing your math skills, to actually think about other things. You could be nearly the smartest math type guy anyone you meet know, and still not accomplish if math is not the key to your chosen subject.

It's interesting, actually. You're motivated by other peoples' low opinions of you -- this pressure you feel in your gut to prove Caledonian et al wrong -- so you've taken that is probably fairly standard human machinery and tried to do something remarkable with it.

My question is, are you still motivated by the doubt you feel about your native abilities, or have you passed into being compelled purely by your work?

Perhaps the truly refulgent (before they had so become) reached a progression tipping point at which they realized (right or wrong, ironically) that they were essentially beyond comparison, and hence stopped comparing.

Then they could allocate the scarce resources of time and thought exclusively to the problems they were addressing, thus actually attaining a level that truly was beyond comparison.

Jaynes was a really smart guy, but no one can be a genius all the time. He did make at least one notable blunder in Bayesian probability theory -- a blunder he could have avoided if only he'd followed his own rules for careful probability analysis.

You come across as very intelligent when you stick to your areas of expertise, like probability theory, AI and cognitive biases, but some of your more tangential stuff can seem a little naive. Compared to the other major poster on this blog, Robin, I'd say you come across as smarter but less "wise", if that means anything to you. I'm not even a huge fan of the notion of "wisdom", but if there's something you're missing, I think that's it.

If you haven't read it, Simonton's Origins of Genius draws a nice distinction between mental agility and long-term intellectual significance, and explores the correlation between the two. Not a terribly well-written book, but certainly thought-provoking.

@EY: We are the cards we are dealt, and intelligence is the unfairest of all those cards. More unfair than wealth or health or home country, unfairer than your happiness set-point. People have difficulty accepting that life can be that unfair, it's not a happy thought. "Intelligence isn't as important as X" is one way of turning away from the unfairness, refusing to deal with it, thinking a happier thought instead. It's a temptation, both to those dealt poor cards, and to those dealt good ones. Just as downplaying the importance of money is a temptation both to the poor and to the rich.

How could the writer of the above words be the writer of today's post? Apparently (as I'm told) you knew from the days of the Northwestern Talent Search that you weren't the smartest of those tested (not to mention all those who were not tested), but certainly one of the smartest. Apparently, you were dealt a straight flush to the king, while some in history received a royal flush. What difference does it make whether someone thinks you are the smartest person they have known, unless you are the smartest person? Does a straight flush to the king meet the threshold required to develop a method for "saving humanity"? If not, why aren't you in the camp of those who wish to improve human intelligence? awaits clap of thunder from those dealt better hands

Eliezer, I've been watching you with interest since 1996 due to your obvious intelligence and "altruism." From my background as a smart individual with over twenty years managing teams of Ph.D.s (and others with similar non-degreed qualifications) solving technical problems in the real world, you've always struck me as near but not at the top in terms of intelligence. Your "discoveries" and developmental trajectory fit easily within the bounds of my experience of myself and a few others of similar aptitudes, but your (sheltered) arrogance has always stood out. I wish you continued progress, not so much in ever-sharper analysis, but in ever more effective synthesis of the leading-edge subjects you pursue.

How much do you worry about age 40? Is that just based on your father? Conway passed 40 before Marcello was born.

If not, why aren't you in the camp of those who wish to improve human intelligence?

I'll take this one because I'm almost certain Eliezer would answer the same way.

Working on AI is a more effective way of increasing the intelligence of the space and matter around us than increasing human intelligence is. The probability of making substantial progress is higher.

Wow, chill out, Eliezer. You're probably among the top 10, certainly in the top 20, most-intelligent people I've met. That's good enough for anything you could want to do. You are ranked high enough that luck, money, and contacts will all be more important factors for you than some marginal increase in intelligence.

First, same question as Douglas: what is it with the brick wall at 40?

Second: This is another great post, its rare for people to expose their thoughts about theirselves in such an open way. Congratulations!

Regarding your ability, I'm just a regular guy(studied Math in college) but your writings are the most inspiring I've ever read. So much self-reflection about intelligence and the thinking process. The insight about how certain mental processes feel is totally new to me. You have helped me a lot to identify my own blind spots and mistakes. Now I can look back and see exactly where I did go wrong in the past and I see with clarity where there was once confusion. I wish I've read this stuff when I was still 13 years old, maybe this could have prevented a lot of the mistakes I did later in life.

Also one of the things I learned from you is that hard work can substitute for intelligence. Think of evolution, even a stupid person can accomplish great things if he bangs his head long enough against the problem. Well, there is still the need of a basic level of intelligence, but I guess you have that.

Did you read Richard Feynman's biography? AFAIK he was also not the smartest and had moments of great self-doubt in his career where he even thought of giving up. I think this turned out to be a blessing because it forced him to visualize things in a more intuitive manner if my recollection is correct. Hence the invention of the Feynman diagrams.

Regarding College, well I went to one and it was one of the biggest wastes of time in my life, together with school. I wish I had been as smart as you and left school at the age of 12.

I second Robin's comment.

A friend of mine, Steve Jordan, once asked me just how smart I thought he and I were. I answered that I think that no-one is really as smart as the two of us both think we are. You see, for many many people it is possible to choose a weighting scheme among a dozen or so factors contribute to intellectual work such that they are the best. You simply define the vector to their point on the "efficient aptitude frontier" as "real intelligence". A dozen or so people associated with this blog and/or with SIAI and a smaller number who aren't appear to me to be on points of the "known to Michael Vassar efficient aptitude frontier", though not necessarily equally mission-critical points. For my "save the world dream team" I would pick a 25-year-old Steve Jobs over a 25-year-old Terrance Tao, though I'd like both of course.

Manuel, "enroll in a grad program for AI" != "you're smart, you should go to college".

Kragen, the short answer is, "It's easy to talk about the importance of effort if you happen to be Hamming." If you can make the ante for the high-stakes table, then you can talk about how little the ante counts for, and the importance of playing your cards well. But if you can't make the ante...

Robin, it's not blind faith in math or math for the sake of impressiveness, but a specific sense that the specific next problems I have to solve, will require more math than I've used up to this point. Not Andrew J. Wiles math, but Jaynes doesn't use Wiles-math either. I quite share your prejudice against math for the sake of looking impressive, because that gets you the wrong math. (Formality isn't about Precision?)

Ken, it's exclusively my work that gives me the motivation to keep working on something for years, but things like pride can give me the motivation to keep working on something for the next minute. I'll take whatever sources of motivation I can get (er, that aren't outright evil, of course).

Douglas, yes, my father changed at 40. But one of my primary sources of hope is that people have been known to do basic research later than this if they changed fields late in life, which suggests that it actually can be a matter of approach/outlook/methodology and avoiding serving on prestigious committees.

Retired, I don't understand the apparent contradiction you see. I participated in the Midwest Talent Search at a young age (not "Northwestern" anything, maybe you're confusing with Northwestern University?) and scored second-best for my grade category, but at that point I'd skipped a grade. But I think I can recall hearing about someone who got higher SAT scores than mine, at age nine. That would be decisive, if the SAT were a perfect noiseless measurement of ability to work on AI.

Vassar: You see, for many many people it is possible to choose a weighting scheme among a dozen or so factors contribute to intellectual work such that they are the best.

Yes, this is the well-known phenomenon where asking someone "How dumb are you?" produces a different answer than "How smart are you?" because they recall a different kind of evidence. But the question I'm trying to answer is "How much potential do you have to solve the remaining FAI problems you know about?" As I said to Robin, I do think this is going to involve taking a step up in math level.

To all commenters who observed that I don't seem to stand out from 10 other smart people they know, either you didn't comprehend the entirety of today's post, or you have very high confidence that you occupy the highest possible rank of human ability.

Vassar - your English is encrypted - more an assumption of intelligence than a sign.

EY - I admire your work. Along with Robin this is the best Show in Town and I will miss it, when it stops.

I actually doubt whether you are accomplishing anything - but this does not seem so important to me, because the effort itself is worthwhile. And we are educated along the way.

This is a youthful blog with youthful worries. From the vantage point of age worrying about intelligence seems like a waste of time and unanswerable to boot.

But those are the stones in your shoes.

@Jef Allbright:

Can you be concrete and specific about where Eliezer is or has been arrogant?

"Most intelligent people I've met" is not informative, we need to give quantitative estimates. My estimate is calibrated based on knowing people who passed various screenings, such as math, physics and programming contests (including at international level), test results on screening exams to top universities, performance in hard university courses, people starting to grasp research and programming, etc. Based on population of regions covered by various screenings, and taking age, gender and different background into account, I can approximately rate these people on the "1 in XXX" scale. I'd say that you need to be at a level of 1 in 300 or so to be able to deeply understand any technical field of human knowledge given reasonable effort, and 1 in 100 to be a competent technical specialist. There is a significant difference (which can cash out as, say, 3x speedup at obtaining given level of aptitude) between people who are 1 in 1000 and 1 in 10000. I know too few people beyond 1 in 10000 (about top 30 in a contest over population of 20 million within a 3-year age interval, given average lifespan of 60 and background selection of 1 in 3 top people to enter the contest) to say whether there is a general enough advantage of being there, or if the performance levels off and more rarely occurring extraordinary ability only presents itself on very narrow task, like blasting through programming contests.

People at all levels are stupid at unknown domains, it takes much effort to start demonstrating "raw intelligence" at anything (although in many things skills partially translate between domains). You can't learn to be creative enough if you don't pass a necessary threshold, but on the other hand if you are past it, sufficient effort will make you able to solve any problem other people can solve, although it'll take more time to attain that level and to solve problems once you did that.

The main problem for getting results is that it's very hard to port smart people on a new field of expertise, to convince them to start thinking about something or to actively work on improving their performance in a given domain. So, it seems that the main problem with seeing (or finding) enough brilliant people in any given field or group is not in the rarity of talent, but in the roads they all took, too few of which lead where you look.

People won't risk working on hard important problems or even think too much about exploring which problems could be important, they choose convenient, safe or enjoyable paths, they choose intellectual dynamic, the process, rather than proper understanding of results or appearance. People you hear from are not the smartest there are in a given subject.

I estimate myself to be around 1 in 1000, more specifically a somewhat blinder, slower and faulty-memory version of 1 in 5000 (as I understand, it's not how many other people perceive their limitations). I clearly see the advantages that people with clearer minds get, but as far as I can tell I'm still able to excel at anything if I devote enough attention to it, given enough time. Extraordinary intellectual productivity is a result of taking the right road, which may depend on happenstance beyond your control. Digging yourself from the pit of blind stupidity (relatively), of seeing only a surface level and stopping the investigation there, is the most important thing (which is what the art of rationality is about, not being stupid, using what you've got, while it falls short of understanding intelligence deeper).

From what I've read, I think that Eliezer is somewhere at 1 in 5000 on this scale, given time he devoted to the study of the subjects and results he produced. He stands out in comparison mainly because too few smart enough people engage in the questions he addresses from the same side, and of those who do hardly anybody devoted much serious thought to them and at the same time didn't get lost on a false road. You don't see the absence of talent, but the initial stupidity in unfamiliar domain or entrenched mistakes where there isn't a solid body of knowledge and authority to force them out.

So, I think that his estimate of 1 in 10000-100000 is too high. The problem is more of convincing the right people to work on the problem and pointing them to the right path, rather than of finding the right people at all. Having an introductory text showing the path is a huge asset, so the decision to compose this book might be a fruitful one.

My own potential intelligence does worry me fairly often. I am currently studying to become an engineer and hope to work on some of the awesome ideas I read about on sites like this. The thing is though, I wasted the first twenty third years of my life. I am currently at twenty-five years old and I have been forced to pretty much start from scratch on everything from social skills to education and after two years I think I am making some headway. I am even starting to understand what Eliezer talks about in all these posts and apply it to my own life as best I can. The math still escapes me but I managed to make it through about half of the Bayesian explanation before getting completely and utterly lost. So I think it is certainly possible learn a huge amount of things even after young childhood but it is rather less efficient. I have had to really struggle to get to where I am now. And since Eliezer is one of the big reasons I am so excited about getting into science I would like to attempt to work in a similar field of research. Yep.

Let me give a shout out to my 1:50 peeps! I can't even summarize what EY has notably accomplished beyond highlighting how much more likely he is to accomplish something. All I really want is for Google to stop returning pages that are obviously unhelpful to me, or for a machine to disentangle how the genetic code works, or a system that can give absolute top notch medical advice, or something better than the bumbling jackasses[choose any] that manage to make policy in our country. Give me one of those things and you will be one in a million, baby.

@Roland

I suppose you could google "(arrogant OR arrogance OR modesty) eliezer yudkowsky" and have plenty to digest. Note that the arrogance at issue is neither dishonest nor unwarranted, but it is an impairment, and a consequence of trade-offs which, from within a broader context, probably wouldn't be taken in the same way.

That's as far as I'm willing to entertain this line of inquiry, which ostensibly neutral request for facts appears to belie an undercurrent of offense.

You're probably among the top 10, certainly in the top 20, most-intelligent people I've met. That's good enough for anything you could want to do.

Okay, I realize you're going to read that and say, "It's obviously not good enough for things requiring superhuman intelligence!"

I meant that, if you compare your attributes to those of other humans, and you sort those attributes, with the one that presents you the most trouble in attaining your goal at the top, intelligence will not be near the top of that list for you, for any goal.

Eliezer,

what's with the ego?

In other words - why are you so driven?

I gather from your posts that you have metaphysical views which make you believe that solving the FAI problem is the most important thing you should be doing.

But is it really that important that you are the one to bring this work to fruition?

Do you think your life will have been unfulfilled, or your opportunity wasted, if you don't finish this, and finish it as soon as you can?

Would building an exceptional foundation, which future exceptional people can improve on, not be achievement enough?

What does it matter how smart you are, if you are doing what you love, and giving it your best effort?

Perhaps it is the fear of being too late that is causing you distress. Perhaps you fear that humanity is going to be destroyed because you didn't build an FAI soon enough. Perhaps you fear that your life will end some 10,000 years sooner than you'd like.

But it is not your responsibility to save the world. It can be fun if you contribute to the effort. But planets are a dime a dozen, and lives are even cheaper than that. We are not really that important. No one is. In the grand scheme of things, our dramas and concerns are lightweight fun.

One of the problems of always being the top banana is that you never learn to realize that you don't have to be the top banana to be fulfilled in your life.

There's no need to worry so much about being on Jaynes's or Conway's level. Do what you do best, and do it because it's fun. If you've been given what it takes, then this is the fastest way to become the master of your field. And even if you didn't have what it takes - which in your case is unlikely - you would still be making a contribution and having fun.

@Jef Allbright:

I suppose you could google "(arrogant OR arrogance OR modesty) eliezer yudkowsky" and have plenty to digest.

Well, I was asking you, not google. But it seems that you are not willing to stand behind your words, making claims then failing to provide evidence when asked. Refering to a third party is an evasive maneuver. Show us your cards!

That's as far as I'm willing to entertain this line of inquiry, which ostensibly neutral request for facts appears to belie an undercurrent of offense.

That's your supposition.

Eliezer, can you clarify what you mean by "You'll note that I don't try to modestly say anything like, "Well, I may not be as brilliant as Jaynes or Conway, but that doesn't mean I can't do important things in my chosen field."

Because I do know... that's not how it works."

Vladimir Nesov: thanks for your comment. I found it insightful.

You say 'That's not how it works.' But I think that IS how it works!

If progress were only ever made by people as smart as E.T. Jaynes, humanity would never have gotten anywhere. Even with fat tails, intelligence is still roughly normally distributed, and there just aren't that many 6 sigma events. The vast majority of scientific progress is incremental, notwithstanding that it's only the revolutionary achievements that are salient.

The real question is, do you want Friendly A.I. to be achieved? Or do you just want friendly A.I. to be achieved by YOU? There's no shame in the latter one, but the preclusion of the latter speaks little about progress towards the former (which I happen to think this blog is immensely valuable towards).

I find myself, except in the case of people with obvious impairments, completely unable to determine how intelligent someone is by interacting with them. Sometimes I can determine who is capable of performing specific tasks, but I have little confidence in my ability to assess "general intelligence".

To some extent, this is because different people have acquired different skills. Archimedes of Syracuse may have been the greatest mathematician in history, but he wouldn't be able to pass the exams in a high school calculus class. Obviously, the reason he couldn't solve these math problems is not that he isn't as intelligent as today's high school students. It's because he never had a calculus textbook.

If you had two black boxes, one of which contained a 14-year-old who scores in the 98th percentile on IQ tests, and the other contained the median college graduate with a degree in some technical field, such as electrical engineering, which black box would appear more intelligent?

It's hard to tell the difference between someone who is actually smarter and someone who has simply learned more. One thing that I learned how to do very well, which contributed greatly to much of my academic success, is translate "word problems" into mathematical equations. There's a systematic way to do this that works on just about any (reasonable) textbook, and it's a task that that I found many of my fellow high school students having trouble with in my science classes.

To what extent is "intelligence" simply a matter of having already learned the best ways to learn?

Also...

I believe that you don't really understand something until you can explain it to someone else, and have them understand it, too.

There's basically two reasons to get called arrogant. One is acting like you're better when you aren't. The other is refusing to politely pretend that the inferential chasm is small. Given where E is and where the mass of humanity are, if I had to make blind-guess assignments for 100 accusers picked at random, and I assigned them all into the "inferential distance" bin, I don't think I'd be wrong once. So, a person asking to be put, or to put some accuser into the "undeserved airs" bin, had better show some sharp evidence!

"Math is a game for the young."

"Perhaps it is the fear of being too late that is causing you distress. Perhaps you fear that humanity is going to be destroyed because you didn't build an FAI soon enough. Perhaps you fear that your life will end some 10,000 years sooner than you'd like."

Humanity's alleged demise is not the only possible way he could be too late. I wonder where Eliezer would turn his attention if someone (or some group) solved the problems of FAI before him.

Eliezer has written a number of times about how comparing your intelligence and rationality to those around you is pointless (e.g. it's never good enough to be good in comparison, etc.). This philosophy has thus far been directed at comparing one's self to lower levels of cognition - but I don't see why it shouldn't work bottom up also. Learn from the levels above you, but do not lionize them. As we all aspire to embody the higher levels, I'm sure Jaynes must have also (an old vampire, but not old enough).

Eliezer: I don't think we should worry about our particular positions on the bell curve, or set goals for where we want to be. Don't fret over the possible limitations of your brain, doing so will not change them. Just work hard and try your best, always attempt to advance - push the limitations. Jaynes was struggling against his meat-brain too. It's human - you both surpassed the village idiots and college professors, now the difference in levels becomes more and more negligible with each step approaching the limit. Everybody is working with meat designed by the "idiot god". Push it to the limit, hate the limit, but don't be self-conscious about it.

We all wish we had gotten an earlier start on things. The importance of them is perhaps something you have to learn as you grow.

Eliezer: It seems to me that uncertainty about your abilities is dwarfed by uncertainty about the difficulty of the problem.

Doug S: The median college graduate in a technical field probably would test in the 95th percentile on most IQ tests and at the 98th percentile on tests weighted heavily towards non-vocabulary crystalline g

Eliezer: Not sure to what extent this helps or answers your questions, but I increasingly as of late find that much of my current "cached wisdom" seems to be derived from stuff you've said.

As far as as actually finding the next generation or whatever, maybe some people here that know how ought to start some "private school for the gifted" that explicitly is meant to try to act almost like a Bayes Dojo or whatever and otherwise train up people in really precise thinking?

While Conway has a huge jump on you in mathematical ability, and I'm pretty sure you're not going to catch up to him, rest assured that you are not strictly dumber than Conway in every respect.

You should bear in mind how the statement "Maybe anything more than around one standard deviation above you starts to blur together, though that's just a cool-sounding wild guess" might apply to me. If your guess is literally true, then, because math is my strong-suit, high mathematical ability is the smartest kind of smart that I can detect at all. For me, philosophical ability and the like would blur into "go to college"-land sooner.

In terms of philosophical intuition, you are head and shoulders above Conway. Remember Conway's "Free will theorem" (a brilliant piece of math to be sure, but very misleadingly named.) Yet, you report never having been confused about free will. My sense of awe at your philosophical intuition has only increased after reading the overcoming bias posts. It's doubly impressive to me, because I keep realizing that you are making explicit more of the helpful little nudges you gave me over the course of our work together, and I am impressed at how helpful some of these things were in practice, and your ability to communicate things which seemed so elusive so clearly. I'm not sure how much of that was native intelligence and how much was starting with a good ideas in your mental toolbox, but I could ask the same thing about Conway.

Eliezer: Look on the bright side, you haven't yet relegated yourself to being a mere administrator and occasional sounding board for others' AI research projects! Ego subjugation is a bitch, but it can have minor rewards of self-satisfaction when actions driven by pressure-free buckshot mental synthesis actually bear fruit. I don't envy that it's of no help to you that the luxury of being carefree relies on the knowledge that smarter people are doing the heavy lifting, and today you're at the top tier of that brain chain!

Maksym: We actually do need someone to translate all this OB stuff very badly, though maybe it's desirable to wait for the book. Still, someone should be presenting it. As for convincing smart college students, there are three fairly separate barriers here, those to rationality, those of information and those to action. I recommend working on barriers to rationality and action first and in conjunction, belief second, and let people find the info themselves. Politics is the natural subject to frame as rationality. Simply turn every conversation where politics comes up into an opportunity to discourse on OB. Rules of etiquette are weak at Harvey Mudd, so this should be OK.

Denis: In technical fields? If so, I unhesitatingly deny the data. I suggest you look at Gottfredson. Lynn is far from trustworthy, but may also be summarizing. Do you really think that people who can't pull 600 on the SAT Math can do engineering?

Dude, you honestly make me ill sometimes. You spoke nothing of the circumstances that got these people to where they are or where they came from. There are people just as "sparkly" and some smarter than these people who have not had the opportunity that these people have. You are blinded by your arrogance and are locked in the present time. You are a smart guy, but you would have a lot to gain in building interpersonal wisdom.

The sparkle you describe is meaningless; non-sparkling borderline-autistic types do just as fine work as the most invigoratingly sparkling individuals. I choose to sparkle through my work, in quiet solitude, not through swaying my limbs excitedly, motor-mouthing like a sports commentator on amphs.

Its a benefit for me to read this post having not read your others, because I can give you an untainted view of it. You are too concerned with intelligence. As long as you stay in this state, you are unusable, and pass up opportunities on becoming usable.
Snap out of it. Accept that there are more intelligent people than you, and they are not flailing, they just get on with it.

Again, I have difficulty understanding why so many people place such a high value on 'intelligence' for its own sake, as opposed to a means to an end. If Eliezer is worried that he does not have enough mathematical intelligence to save the universe from someone else's misdesigned AI, than this is indeed a problem for him, but only because the universe will not be saved. If someone else saves the universe instead, Eliezer should not mind, and should go back to writing sci-fi novels. Why should Eliezer's ego cry at the thought of being upstaged? He should want that to happen if he's such an altruist.

I don't really give a damn where my 'intelligence' falls on some scale, so long as I have enough of it to accomplish those things I find satisfying and important TO ME. And if I don't, well, hopefully I have enough savvy to get others who do to help me out of a difficult situation. Hopefully Eliezer can get the help he needs with fAI (if such help even exists and such a problem is solvable).

Also, to those who care about intelligence for its own sake, does the absolute horsepower matter to you, or only your abilities relative to others? IE, would you be satisfied if you were considered the smartest person in the world by whatever scale, or would that still not be enough because you were not omniscient?

Of course I want there to be someone smarter than me to take over, from an altruistic perspective. Or even from just a selfish perspective of being scared, wanting a vacation, and feeling a bit isolated.

And of course if that actually happened, it would be a severe blow to my ego.

And so long as I can do the expected-utility-maximizing thing and invest the appropriate amount of effort into preparing for the possibility without betting the whole farm on it, I have no intention of hacking at my emotions on either score.

I know how you feel, in a couple ways. My high-school guidance counselor looked at my middle school transcript and told me I might realistically aspire to go to a UC school (as opposed to a school in the Cal State system). (I ended up going to Harvard and Caltech.) On the other hand, the year I finished my Ph.D. (at the age of 29) one of my college acquaintances, a brilliant mathematician, became one of the youngest full professors in the history of Princeton University, and when my Ph.D. advisor was 29 he had already been a professor at Caltech for several years (after graduating top of his Caltech class, finishing a Ph.D. in three years at Cambridge on a Marshall Scholarship, and doing a postdoc at the Institute for Advanced Study).

Though I know I'm not (alas) at the highest level, I feel fortunate to be smart enough to know how smart I'm not.

N.B. The introduction contains a small non sequitur unless you happen to know the book's author. Here's a quick edit:

I once lent Xiaoguang "Mike" Li my copy of Probability Theory: The Logic of Science by E.T. Jaynes.

It's a general rule, I've observed, that you can't discriminate between levels too far above your own.

Do other people agree? If so, what do you propose distinguishes between intelligence/mathematical ability and athletic ability?

If so, what do you propose distinguishes between intelligence/mathematical ability and athletic ability?

It is possible for a person to produce an accurate evaluation of a subset of their own intellectual skills, but certain skills cannot be evaluated, because presumptions about those skills are required for the evaluation to take place. You should not ask questions about subjects in which you presume you already know the answers, and you cannot ask questions about subjects where answers must be presumed in order to be able to ask at all.

Lara, I don't think they value it "for its own sake" as opposed to as a means to an end; rather, they see it as a necessary condition for achieving their ends, and are worried they don't have what it takes. Nothing but an anxiety trip.

And of course, there's also the ego thing -- when people build superiority over others into their self-image. This is counterproductive, of course. When someone else demonstrates that they're "smarter" than you by offering unexpected insight, you don't fatalistically wallow in jealous misery; you listen to the content of what they say, in the hope of becoming as smart as they are.

Eliezer of all people ought to realize this (actually I suspect he does).

FWIW, I've met both Eliezer and John Conway, and have spent approximately the same total amount of time with both of them (on the order of 10 hours). I don't know which of them is smarter. Yet I suspect neither is too far above my own level for me to be able to e.g. benefit from listening to a conversation between them.

Eliezer, Komponisto,

I understand the anxiety issues of, 'Do I have what it takes to accomplish this..."

I don't understand why the existence of someone else who can would damage Eliezer's ego. I can observe that many other people's sense of self is violated if they find out that someone else is better at something they thought they were the best at-- the football champion at HS losing their position at college, etc. However, in order for this to occur, the person needs to 1) in fact misjudge their relative superiority to others, and 2) value the superiority for its own sake.

Now, Eliezer might take the discovery of a better rationalist/fAI designer as proof that he misjudged his relative superiority-- but unless he thinks his superiority is itself valuable, he should not be bothered by it. His own actual intelligence, afterall, will not have changed, only the state of his knowledge of others' intelligence relative to his own.

Eliezer must enjoy thinking he is superior for loss of this status to bother his 'ego'.

Though I suppose one could argue that this is a natural human quality, and Eliezer would need to be superhuman or lying to say otherwise.

I have no idea if it's a natural human quality. It's surely one of my qualities. It's not that I would permit my mind to think verbal thoughts like "How good it is to be above others." But there's a zest in being the best. It feels good to complete a difficult race and it feels good to win a gold medal; they are separate, different good feelings. I can imagine people who would only care about having completed the challenge, but they wouldn't be me.

Since my mind doesn't want whatever I choose it to want, I accept that both desires are a part of me and that both desires keep me motivated to continue studying. Though even the desire to solve hard problems for yourself, is not without its dangers.

Doesn't fit the stereotype of Deep Wisdom, I know. I would be prouder if I was Gandalf, because that would be, you know, cool. But you see, this isn't about my pride.

I can imagine people who would only care about having completed the challenge, but they wouldn't be me.

I'm not sure there are any people like this who are capable of occasionally winning. OTOH, the prospect of never winning might force someone to rationalize themselves into this position.

The proof is in the math and/or in the protopudding, is it not? There are people/groups who already have either or both. If you have neither, what's your sense of relative achievement/skill/IQ based on?

What (math &/ prototype) do you have? If none, what do you plan to have, when? It seems you'd have to blaze past those who already have their stuff out in the real world behaving ever more AGI-ishly by the day, to meet your criteria for success. A tall order to be sure.

Some else wrote

"
This is a youthful blog with youthful worries. From the vantage point of age worrying about intelligence seems like a waste of time and unanswerable to boot.
"

and I find this observation insightful, and even a bit understated.

Increasingly, as one ages, one worries more about what one DOES, rather than about abstract characterizations of one's capability.

Obviously, one reason these sorts of questions about comparative general intelligence are unanswerable is that "general intelligence" is not really a rigorously defined concept -- as you well know! And the rigorous definitions that have been proposed (e.g. in Legg and Hutter's writing, or my earlier writings, etc.) are basically nonmeasurable in practice -- they're only crudely approximable in practice, and the margin of error of these approximations is almost surely large enough to blur whatever distinctions exist between various highly clever humans.

I have no doubt that you're extremely smart, and especially talented in some particular areas (such as mathematics and writing, to give a nonexhaustive list) ... and that you're capable of accomplishing great things intellectually.

As an aside, the notion that Conway, or von Neumann or any other historical math figure is "more intelligent than Eliezer along all dimensions" seems silly to me ... I'm sure they weren't, under any reasonable definition of "dimensions" in this context.

To take a well-worn example: from my study of the historical record, it seems clear that Einstein and Godel were both less transparently, obviously clever than von Neumann. My guess is that von Neumann would have scored higher on IQ tests than either of those others, because he was incredibly quick-minded and fond of puzzle-type problems. However, obviously there were relevant dimensions along which both Einstein and Godel were "smarter" than von Neumann; and they pursued research paths in which these dimensions were highly relevant.

"General intelligence" has more and more meaning as one deals with more and more powerful computational systems. For humans it's meaningful but not amazingly, dramatically meaningful ... what's predictive of human achievement is almost surely a complex mixture of human general intelligence with human specialized intelligence in achievement-relevant domains.

Pragmatically separating general from specialized intelligence in oneself or other humans is a hard problem, and not really a terribly useful thing to try to do.

Achieving great things seems always to be a mixture of general intelligence, specialized intelligence, wise choice of the right problems to work on, and personality properties like persistence ...

-- Ben G

Achieving great things seems always to be a mixture of general intelligence, specialized intelligence, wise choice of the right problems to work on, and personality properties like persistence ...

With a pinch of being in the right place and the right time, bake on 350 for 10-30 years.

Ben,
I kind of disagree with you. First, what we call "general intelligence" is itself a form of specialized intelligence: specializing optimizing successful outcomes in real time in our apparent reality. so the mix you recommend in "achieving great things" would itself be "general intelligence", not general intelligence plus something else (other than luck).

Since most people who "achieve great things" seem to me to be playing life at least in part as a poker game (they don't seem to put all their cards out on the table) I think outcomes may be a better measure than propaganda. "Increasingly, as one ages, one worries more about what one DOES, rather than about abstract characterizations of one's capability." I'm not sure that comes from wisdom, rather than the rationally adjusted propaganda of an older person (look at my status enabled institutional power to achieve) contrasted with that of the younger person (look at my superior capabilities, with a brain at its physical prime).

Increasingly, as one ages, one worries more about what one DOES, rather than about abstract characterizations of one's capability.

This definitely happened to me. Between the ages of about 10 - 14, I was utterly obsessed with finding out what my IQ was. Somehow, somewhere along the way, I'd picked up the notion that Smartness in quantity was the most important thing a person could possibly have.

And it drove me frankly batty not knowing how much Smartness I had, because (a) I was insecure and felt like I needed to find out I had a "high enough" number in order to permit myself any sense of self-worth, and (b) I had an idea fixed in my mind that only "geniuses" with IQs 150 or above could have any hope of addressing any of the interesting questions and topics that dominated my thoughts as a geeky little kid: faster-than-light travel, Grand Unified Theories, etc.

I spent a lot of time trying to find any papers/reports/test scores my parents might be hiding away, hoping that I'd be able to discover through doing this some idea of the quantitative value stamp I was convinced must be on my brain somewhere (though not directly viewable by me).

I didn't actually find any of these papers until I was in my late teens, and by then I found with some surprise that I didn't care all that much what they said. At some point between the ages of 14 and 17 I'd managed to get over my IQ obsession and move toward a different brain-related obsession (one considerably less worry-inducing): that of how brains, and in particular mine, worked at all. And in ceasing to be obsessed with quantitative test-based measurements, lo and behold, I found it far easier to actually think about things and just plain learn.

I do now know what my age-4 Weschler score was, and it wasn't 150. Not even close. I took another Weschler (the adult scale) in college, and while that score ended up being quite a bit higher than my age-4 score, it was still lower than I'd originally hoped it would be. But it didn't matter to me in the least from an emotional standpoint by then, because I'd already managed to accomplish things (like getting an A in calculus) that I'd have considered the province of people with far higher IQ scores than I actually had. Not to mention the fact that when I looked at my subtest scores, they were all over the map -- I had a higher than average Block Design, but lower than average Picture Arrangement, for instance.

At this point I tend to see IQ (at least as measured on tests) as being very limited in terms of what information it actually tells you about what someone is capable of doing. E.g., I don't think IQ scores can definitively tell you when someone is going to "hit a wall", so to speak, in terms of what mathematical theorem they will absolutely get stuck on when they encounter it (or what engineering problem they might be able to solve, etc.).

It almost seems like some of these posts are suggesting a desire for much greater predictive ability than any test or ten-minute impression could possibly actually reveal in something as complex and feedback-sensitive as a human individual. And while I'd like as much as anyone for the world and everyone in it not to be destroyed (whether in one great cataclysm or a gradual tragic fade-out), I've come to terms with the fact that, as corny as it sounds, all we can do is our best, and we must do this in the utter absence of perfect knowledge regarding the limits of our individual or collective capacity.

Eliezer, don't think to yourself that you only have until you are 40. As somebody else noted and you didn't acknowledge, Marcello was not yet born when Conway passed 40. You mentioned your father, and I don't know the specifics, but surely you know that plenty of people have done great work, sometimes their best, past 40, and that with every passing year, due to advances in health, medicine, etc., "youth" extends further and further into our life.

And as another poster mentioned, I have almost no doubt that Von Neumann would have blown Einstein (possibly Godel) out of the water in terms of ability, and at the age of 10 or 11, William James Sidis would have smoked them all (at comparable ages). Even if there is such a thing as general intelligence and even if one has less of it than somebody else, it is still very much possible that one is vastly more talented in particular areas.

Einstein probably didn't have an IQ in the realm of Von Neumann or Newton or Sidis, but what he did have was an incredible physical intuition and the ability to think deeply about the philosophical underpinnings of physics in a way that few others could (or can). As others have said, even if Conway actually is smarter (which you should not just blindly accept, because we all have cognitive biases that might prevent us from making an accurate judgment), it doesn't mean that you don't have profound gifts that will end up being the deciding factor in solving FAI or whatever else you set your mind to.

Looking at Von Neumann at age 24 and Einstein at age 24, NOBODY would have thought it remotely possible that Einstein would be the one who would achieve enough in his field to be considered a successor to Newton. Von Neumann, though he accomplished more than 100 "mere" geniuses could ever hope to accomplish, cannot be said to be a successor to (or in the same league as) a Gauss or somebody of similar stature.

Since you're so fond of quotes, here's one for you:

If you can solve your problem, then what is the need of worrying? If you cannot solve it, then what is the use of worrying?

- Shantideva (Indian Buddhist philosopher)

Just get on with it. Self-doubt and endless self-analysis do not contribute to the end goal.

Supporting Ben Goertzel's comment:

Michael Shermer revised his book, Why People Believe Weird Things, to contain a chapter called “Why Smart People Believe Weird Things”. In it, he quotes studies by Hudson, Getzels, and Jackson showing that “creativity and intelligence are relatively orthogonal (i.e., unrelated statistically) at high levels of intelligence. Intuitively, it seems like the more intelligent people are the more creative they will be. In fact, in almost any profession significantly affected by intelligence, once you are at a certain level among the population of practitioners (and that level appears to be an IQ score of about 125), there is no difference in intelligence between the most successful and the average in that profession. At that point other variables, independent of intelligence, take over, such as creativity, or achievement motivation and the drive to succeed.”

Actually RU, that's a good approximation for many/most professions, but not all that good an approximation.
http://www.vanderbilt.edu/Peabody/SMPY/DoingPsychScience2006.pdf
gives more detail, showing a significant marginal impact from, at the least, 99.99th percentile math achievement at age 12 relative to merely 99.8th percentile math achievement at age 12.

Is this study talking about Nobel Prize winners - or better yet, Fields Medal-winning mathematicians? Or just authors or something? I'm about ready to say "I defy the data; what about von Neumann?" Maybe there are people who can achieve through diligence what others achieve by genius, but to say that genius doesn't help at all... I defy the data.

(If you told me that IQ didn't make a difference past 140, I'd be quite willing to believe that IQ tests don't work past 140. Richard Feynman's measured IQ was 137, which as John K Clark observed, says more about IQ tests than it does about Feynman.)

Feynman's measured IQ was 123, not 137. And we already know that IQ tests do not measure vitally important aspects of cognition -- in Feynman's case especially, he was quite strong in those aspects while being weak in the aspects measured. (At least, I know that. What the rest of you know is less certain.)

This is one of the primary reasons why people who think we can use IQ scores as a representation for the higher-level aspects we can't measure well (because they're supposedly correlated with IQ) are wrong. (I'm looking at you, Vasser.)

IQ tests do not measure synthetic capacity, imagination, creative potential, or self-restraint / the ability to inhibit low-level drives and impulses. They measure only the ability to complete certain atomic functions in a limited subset of cognitive tasks. That makes them useful -- extremely so -- but not definitive. Not even close.

There's another aspect of the shortcomings of IQ tests that people might not be aware of. Cognition is quite flexible, and abstract problem-solving ability can be met by many combinations of underlying, modular capacities. A person lacking in certain respects can make up for the lack, at the price, perhaps, of thinking a little more slowly.

Take me for an example. On the WISC-III IQ test, my combined score is 145. There are two composite scores that the combined score is made up of, the verbal score (I got 155, the maximum possible on that test) and the performance score (I got 125). There are also a number of different individual capacity scores. On most, I scored above the 95 percentile. On two or three, I scored right in the middle, and in one (visual short term memory) I scored in the first percentile.

Let me repeat that. I scored in the first percentile for the capacity to keep visual information in my short-term memory. (I scored in the 97th for aural short term memory, and 99.9th for linguistic.) How does that change how I solve problems, how I think about the world? Well, I perform many tasks about twice as slowly (but just as accurately) as others with my composite IQ. I have to use other circuits than most people do to solve the same problems, circuits that aren't as efficient. Circuits that may even work slightly differently, giving me a different perspective on problems, which may be superior or inferior, I don't know (likely depending on the individual problem). I strongly suspect that this is a large part of the cause of my intense dislike of school.

(BTW, people with a large difference between performance and verbal IQ are classified as having non-verbal learning disorder. That's right, even really smart people can have learning disorders.)

IQ is not a single number. Even IQ recognizes a large part of the complexity of human intelligence. It's not the psychologists that make the mistake of reducing it to a single number.

Why did I write this long comment on a dead thread? Dunno.

Gentlemen - Let me propose that the heart of serious intellectual achievement is synthesis, creativity, simplicity.

These are factors that actually increase with age and are not "IQ" or "g" driven. In fact I believe Edward de Bono argued that creativity drops at IQ 125 or so: maybe because people begin to fall into an "expert trap," where they have to maintain their previous work and expert status more than anything else.

Creativity need not decline with age at all - if you can avoid common habit errors.

My objection to Vassar is just that all these "tests" are highly flawed and biased - they consistently disfavor certain people and favor others. They just do, sorry, and this alone invalidates them or at least diminishes their usefulness.

My other comment to you all has to do with Feynman. I once asked another member of the Project, who is a famous emeritus experimentalist, about him. He told me that what distinguished Feynman was his wit and curiosity about things that others didn't think were "on the critical path," so to speak. Wit and curiosity are completely untestable, but if you look at real achievers I believe you'll find these qualities extremely important.

The courage to appear silly to avoid the expert trap - wit - careful avoidance of habit error - constant search for bias - doubting intuitions - deliberately slowing down to allow more time for divergent thought esp. if you are overclocked - synthesis - simplicity - tenacity - the person with these 9 qualities will be a thinker for the ages.

I think you're on the right path, frelkins, but this?

all these "tests" are highly flawed and biased - they consistently disfavor certain people and favor others.

How does the latter follow at all? If we had a test that measures everything you think constitutes real intelligence, it would consistently disfavor certain people and favor others. It would disfavor stupid people and favor smart people. That's the point of an intelligence test.

I don't believe IQ tests measure everything. There's a certain feeling when being creative, and when completing these tests I have not felt it, so I don't think it's measuring it.

Also I am not sure intelligence is general. At the level of ordinary life it certainly is, but geniuses are always geniuses at something, e.g. maths, physics, composing. Why aren't they geniuses at everything.

Does anyone have a reputable source for Feynman's 137? google makes it look very concentrated in this group, probably the result of a single confabulation.

Sykes and Gleick's biographies both give 12x. Sykes quotes Feynman's sister remembering sneaking into the records as a child. This seems important to me: Feynman didn't just fabricate the 12x.

Math smarts are not the most important thing. Basic reasoning skills are vital (even if they are based on heuristics that are sometimes wrong), management skills are extremely important, intelligence augmentation skills are a must, touchtyping is very useful, etc.

Overall you should think not in terms of competitiveness (whether you are smarter than everybody else), but in terms of co-operation (how you can complement others, how they can contribute their skills to complement yours).

And for the record, I don't think you are the smartest person I know (although you are very smart). I suspect that I may have a better skillset than you do. :)

Since this is now kinda on-topic... I don't think Eliezer Yudkowsky is considerably more intelligent that I am. I'm aware of Dunning-Kruger effect, but the interesting part is that I simply don't find any way to overcome this. I'm fairly intelligent, but since people around here regard my barely-MENSA(probably not even that) -level of IQ a minium requirement to even read this blog, the situation I'm in is fairly interesting. I see repeated claims of super-intelligence, but I can see just someone who has had few more years to hone his skills and who has wasted less years on doing pointless things.

So, I'm kinda curious: What's it with Elizer Yudkowsky that makes everyone look up to him? I see some fairly interesting all-around posts, but is it that you see something more? I think there is a possibility here that the reason you admire Yudkowsky and I don't is partially because he Seems Deep(in the sense that he makes sense immediatly but seems novel), while I spent my earlier life discovering much the same stuff alone. It lacks novelty(to me), but adds details and strengths which I attribute to experience, and thus I judge Eliezer less "deep", and, consequently, less superhuman? This possibility gains some credibility from the fact that in my own little circles, I have pretty much the same sort of reputation as Eliezer has here(and, I think this is surprisingly much about personality, and thus I assume it's surprisingly rare for people here to be have that reputation), and before I knew of Eliezer, I had plans of becoming something much like he is now.

I presented two possibilites: Should I accept that I'm just incapable of distinguishing the level so much above my own, or should I defy the public opinion and regard Eliezer as "not that smart", because it's more about personality and "seeming deep" than about real difference in mental machinery. However, third option exists: I haven't read the stuff that makes everyone admire Eliezer so.

I'm really interested in manifestations of intelligence, so this issue is of a great importance to me. Especially if it is about Dunning-Kruger, I wanna understand how to overcome that. Maybe it's just that I pass those "technical-seeming" parts that actually demonstrate amazing intellectual stunts, and I should make a better mental notes every time I'm forced to skip some phrase without completely grasping the meaning.

a friend of mine thought this was relevant: “Mediocrity knows nothing higher than itself, but talent instantly recognizes genius.” - Conan Doyle

I find the idea that there are a lot of more intelligent people in the world than me comforting, especially in my chosen fields. Not because I feel this gives me an excuse to slack off and let them do the hard work, but because competition seems to drive me and keep me happier than anything else. Since finding lesswrong and related sites where people discuss AI, programming, and rationality, my efforts have improved considerably. I am far from competing with most of the people here, particularly you, but at least I have mental patterns I can model to improve.

I know people with greater mental horsepower than you, but none of them ever persisted at any problems that are hard enough to test the limits of their abilities.

I doubt that Jaynes became Jaynes by aspiring to a level. Too bad we can't ask him.

Don't despair of surpassing Jaynes. He, and a great many others, have given you a leg up that Jaynes never had. People seem formidable because they're practiced in mental kung fu that you don't know. Darwin is remembered for an idea you can teach an 8 year old today.

I am blogging here for many reasons, but a major one is the vision of some younger mind reading these words and zipping off past me.

Thank you.

I suspect you and Luke do not share a referent for "better philosophy" here. In particular, I doubt either Luke or Eliezer would agree that the ability to write clearly, or to analyze and formulate arguments for purposes of compellingly engaging with existing arguments in the tradition of analytic philosophy, is the rare skill that Luke is talking about.

Trying to have a conversation about how hard it is to find an X without common referents for X is not likely to lead anywhere productive.

You're right, I should say more about what I mean by "Eliezer-level philosophical ability." Clearly, I don't mean "writing clarity," as many of my favorite analytic philosophers write more clearly than Eliezer does.

It'll take me some time to prepare that explanation. For now, let me show some support for your comment by linking to another example of Eliezer being corrected by a professional philosopher.

I'm not claiming that clarity isn't a benefit, and as far as I can tell nobody else is either.
I agree that it's not hard to write "someone who can do philosophy well in the LessWrongian style".
And sometimes one person can miscommunicate all by themselves.

Downvoted because:

• In my experience of philosophy, there are very few philosophers at EY's level.
• You provided no evidence for your claims; and when you edited your comment ...
• ... and the link you gave consists of him clarifying his terminology, and thanking them for interpreting his unclear wording charitably.

Also because it irritates me that this site is scattered with comments at anything from -3 to +15 (not exact figures) that criticize cryonics/ASI/other things lots of us believe in, LW policies, or EY, and then talk about how they're going to get downvoted into oblivion for speaking out against the consensus.

[Edited for formatting.]

In my experience of philosophy,

Which is what?

Mine.

In my experience of philosophy, there are very few philosophers at EY's level. i don;t see why disagreement would merit a downvote. Isn;t that the whole problem?

You made a claim that I - and, you yourself acknowledged, most LWers - had a low prior for. You did not provide any evidence for this claim. That seems worth a downvote, as long as you don't provide a disclaimer that you can't provide your evidence right now or something.

there are very few philosophers at EY's level.

"Level" meaning credibility by normal standards (ie he as good as Dennett, Chalmers, Parfitt..) or "level" meaning he's good at being lessWongian because he invented it?

Well, I was impressed by LW, so there's naturally going to be some correlation.

Level meaning how good he seems at evaluating philosophical arguments, identifying flaws, and mostly avoiding the flaws that seem especially common in philosophy. I'm not sure how abnormal these criteria are, but then normally we're not talking about FAI.

You provided no evidence for your claims;

Not in the first draft. You downvoted a version with evidence.

One piece of evidence, which, as I noted, is at best extremely weak and misleadingly labelled.

and when you edited your comment.

That's incomplete. I hardly need point out the problems of making mistakes when you are pointing out mistakes.

Whoops. that should be an ellipsis. Fixed.

The link you gave consists of him clarifying his terminology, and thanking them for interpreting his unclear wording charitably.

So? Philosophy is mostly arguing, and arguing more clearly is arguing better...is better philosophy. Are you saying the link was irrelevant or what?

To the extent that rhetorical skills are uncorrelated with epistemic skills, yes, I am. He's not trying to persuade the problem, he's trying to solve it.

Also because it irritates me that this site is scattered with comments at anything from -3 to +15 (not exact figures) that criticize cryonics/ASI/other things lots of us believe in, LW policies, or EY, and then talk about how they're going to get downvoted into oblivion for speaking out against the consensus.

ETA: But the fist draft was downvoted! As was pretty well everything I have said on the subjecta

But not, I suspect, simply because you're contradicting the consensus. You have made sweeping claims, not considered likely by most LWers, regarding, among other things, users on this site (which we can all see,) yet you failed to provide examples; your lack of examples is at best unhelpful and at worst ... well dishonest or symptomatic of massive bias, whichever of those is "worst".

Also I am not clear why that is irritating.

Well, they never get called on it, and it promotes misconceptions regarding the site and the karma system specifically.

ETA. But thanks for the comment anyway.

Thank you for replying. I upvoted the reply, incidentally.

X being a surprisingly simple counterargument. Not a new one, necessarily, just one that the text either fails to address or quietly dismisses without actually refuting it.

An annoying amount of philosophy is guys sitting in armchairs going "but what if contradictory things can be true? Didn't think of that, didja?"

EDIT:

For example:

Swampman, original formulation:

"it can't recognize anything, because it never cognized anything in the first place."

(Based on his glass marble thought experiment, Davidson defines "recognize" causally; that is, whatever caused the idea of the marble is what he means by "the marble", even if it has changed shape or been replaced with a copy. A nonstandard but useful definition, which implies we can be mistaken about what we mean/recognize/etc.)

Swampman is, of course, essentially a Boltzmann brain; he was formed by chance into a form that happens to be conscious. But more than that, he has been formed into a copy of someone else. Now, if he was formed entirely by chance, then an observer should not consider his beliefs to be correct, any more than a random number generator can reliably predict the weather. In this sense the concept of "recognize" is working perfectly; the author is just being mean.

However, the reader of course knows that Swampman is causally connected to Davidson, via the author's fiat; thus our intuitions in this case are closer to a clone than a true Boltzmann brain. And if a character in the thought experiment were to notice that Swampman was a perfect copy (which seems reasonable) they would have sufficient evidence postulate the author. Or at least some other god-level power screwing with them. In this sense Swampman is causally connected to Davidson - who really did "cognize" his friends - and thus genuinely "recognizes" them."

And, of course, an unfortunate amount of philisophy magically disappears when you start playing taboo and otherwise avoiding arguments over definitions.

SECOND EDIT:

OK, I can't reply to comments here due to the karma toll. If anyone's interested, I could post the example on an open thread for criticism?

EDIT: there goes another conversation. Thank you karma toll.

You made a claim that I - and, you yourself acknowledged, most LWers - had a low prior for.

i know. But you are not supposed to downvote for disagreement.

I'm not downvoting for disagreement, I'm downvoting for absurd claims without any damn evidence. If you had provided, say, an example of a LW user who is better at philosophy - as opposed to a terminology quibble - then I would not have downvoted even if I didn't think it was sufficient.

Well, I was impressed by LW, so there's naturally going to be some correlation.

Level meaning how good he seems at evaluating philosophical arguments, identifying flaws, and mostly avoiding the flaws that seem especially common in philosophy.

And i would say he isnot good at coming to clear conclusions,

I think you misspelled "correct" there.

examing all the relevant arguments,

I always considered phiosophers far to willing to spend ages dealing with some BS position that could be demolished in seconds if they were trying to win and not just "do philosophy" for as long as possible, but whatever.

aviding known errors,

If you have spotted an error, I advise you to point it out. Not just make vuage statements about how there are totally loads of errors.

avoiding hubris,

Truly, the gods shall punish him for his arrogance.

using standard terminology, etc.

I suppose standard terminology might help people critique his work, at that. Still, it seems a minor flaw TBH.

One piece of evidence, which, as I noted, is at best extremely weak and misleadingly labelled.

I fail to see what was misleading.

You called it EY "thanking real philosophers for fixing his arguments" which, in the context of you saying that there were better philosophers on LW, rather implies that their skill at philosophy allowed them to identify a mistake well known to, well, experts in the field; not them interpreting his unclear terms charitably, which while a good thing is hardly relevant.

So? Philosophy is mostly arguing, and arguing more clearly is arguing better...is better philosophy. Are you saying the link was irrelevant or what?

I think you accidentally quoted your self instead of me there.

"Rhetorical" is a rather rtheorical way of saying "clear". EY can claim to have a Wonderful Solution in his head that no-one can undestand, but thre is no reason anyone else shouild buy that. To solve a probllem is to publish a solution that persuades others, for all practical purposes.

Um, no. To solve a problem is to find the correct solution, which may or may not require persuading others that it is correct, or indeed that it is incorrect or that they shouldn't eat tuna.

But clarity has nothing to do with "persuasion" in the advertiser's or politician's sense. Clarity is not a varnish that makes bad arguments look good: clarification is a major part of the substance of philosophical solution.

Considering how unclear to a layman much philosophy is, I assume you mean that "clarity" is helping experts to find any mistakes you may have made. This is important, and indeed a separate discipline from what we might call the Dark Arts. Eliezer could probablydo with some more of this, yeah. OTOH, philosophers generally seem pretty bad at this, so I'm not sure if Eliezer is unusually bad. And it's certainly not the only or even main thing philosophers are supposed to be good at.

But not, I suspect, simply because you're contradicting the consensus. You have made sweeping claims, not considered likely by most LWers, regarding, among other things, users on this site (which we can all see,) yet you failed to provide examples; your lack of examples is at best unhelpful and at worst ... well dishonest or symptomatic of massive bias, whichever of those is "worst".

I argue my points.

Did you quote the wrong thing here too? I'm not sure how this link is relevant.

You did not provide any evidence for this claim.

ETA:

What would evidence that someone is bad at something look like?

Evidence that someone is unusually bad would look like lots of people being better. So would evidence that "A number of better-than-EY philosophers have visited this site, sometimes displaying their skills in discussions with EY."

Moved due to low karma.

This restriction applies as intended, don't evade it.

http://wiki.lesswrong.com/wiki/Deletion_policy#Prolific_trolls

(You now have minus 250 30-day Karma, so I'll start banning/hiding some of your comments (they will remain accessible from your user page).)

I won't play the definitional games -- you yourself talked about "low-level trolling" which you excused as "teasing", and so you could have used your definition, whatever it is. But you didn't; and instead you avoided promising not to troll or stating that you've not trolled before.

I won't respond to you again, atleast until such a promise has been made, and perhaps not even then.

This link seems not to answer the comment ,:-. is this mistaken or did EY use that fallacy?

Black swans flock. There are any number of things with very low probablity and very high positive and negative payoffs. You can;t control for them all. I suggest renormalisation: hand-wave them away by assuming they cancel out.

Given the fairly uniform negative reaction to your posts, judging by your 30-day karma, you may want to consider looking for a forum where you will be better appreciated.

You accused Eliezer of committing the noncentral fallacy, you did not demonstrate that he committed it.

Eliezer posited a categorization of "trolls" defined by the practice of deliberately pissing people off on the internet, of which people who incite others to commit suicide are an extreme example. If this is the uniting quality of "trolls," then it's not unreasonable to conclude that we don't want any of them in the community, not just the more extreme examples.

Not all diseases will kill you or make you wish you were dead, so it may seem "unfair" to group ones that will with ones that merely, say, make your feet itchy, but that doesn't mean it's unreasonable to prefer to have no diseases.

Surely low enough not to be overcome by you being impressed or you agreeing with his philosophy

"Here is a very simple example of Bayesian reasoning, that most people are in fact capable of. Suppose we draw a random number between 1 and a million; the prior for any particular number between 1 and a million is straightforwardly very low - one in a million, of course. Now, I have just generated the number 493250 using random.org. Surely this prior of 1 in a million that I have generated any specific number like 493250 is low enough to not be overcome by you being impressed by looking at this comment and see '493250' in it? The prior for you having very special powers of perception of the right number is likewise proportionally low to how very special it is, and so on."

"Here is a very simple example of Bayesian reasoning, that most people are in fact capable of. Suppose we are looking at people who write clip art web comics; the prior for any particular clip art being the best or most popular is straightforwardly very low - one in a million, say, or what ever is your number. Now, we look at http://www.qwantz.com/index.php Surely this prior of 1 in a million is low enough to not be overcome by you being impressed by looking at this Dinosaur Comics? The prior for you having very special powers of perception of clip art is likewise proportionally low to how very special it is, and so on."

The ensuring debates and demands for evidence that something with very low prior isn't true, are particularly illuminating with regards to just how incapable certain self proclaimed Bayesians are of the most basic forms of probabilistic reasoning.

Yes. I agree. Some of these self proclaimed Bayesians cannot even fully specify their examples or prove their arguments or explain the crucial part of what they were probably arguing.

I would of thought the latter but I now think you're honest given the earlier conversation involving 'crazy' anthropic reasoning of mine which turned out to be favoured by much everyone at SI as well, contrary to your claims.

When did I claim no one at SI held your views? That would've been hard since you refused to use standard terminology like SIA or SSA which I could then go 'ah yes, that's Bostrom's current view'.

If you had some omniscient Omega that had a web interface where you could enter "Pick an 1 in a million - quality philosopher" and it would reply "Eliezer Yudkowsky" and that's how you came around Yudkowsky, then it would have been analogous to that random.org example.

...Prior for the comic is low. You update it away if the choice of comic is very well correlating with what you consider the "best". If you were just shown various clip art at random you'd have a lot of trouble guessing the most popular one, because your eye for popularity certainly won't provide enough evidence.

No, that's not the analogy. The analogy is that there are at least 2 ways in which we are long past a prior of 1 in a million and don't have judgments which are equivalent to random choice, and those were illustrating them: the first is one's own ability to recognize some level of quality in a philosopher, and the second is about looking at a non-random selection at the end of a process with some selection for quality.

Are you making a point that popularity of a philosopher among non philosophers is very correlated to their philosophical ability?

Even a small correlation is enough to move the needle.

What's about lack of recognition by other philosophers, how is that correlated with philosophical ability? What's about Jesus, a dead philosopher who's quite damn popular?

So you cite, in a statistical claim throwing around numbers like 1 in a million, a single example? And I wonder how many people really consider Jesus a philosopher, as opposed to an excuse like GWB to signal their religion and cover up that they don't actually have any preferences as to secular philosophers...

But yeah, popularity is a meaningful index! Go down the list of great philosophers and you'll find they are popular and even appear in pop culture; Zeno, Plato, Socrates, Aristotle, Confucius, Descartes, Nietzsche, Russell, Wittgenstein to name a few off the top of my head are all widely read by laymen and appear in popular culture, and were often world-famous in their own lifetime. Of course it's not a perfect correlation - not all great philosophers will find popularity after their death among non-philosophers (Plotinus or Spinoza or Hume may have been the greatest philosophers of their time but only philosophers read them these days) - but think of how many minor or poor philosophers from those respective time periods remain obscure... Very few of them succeed like Ayn Rand in being a poor philosopher and also popular.

If you don't know it other than by name, that's not my problem. It was straightforward mathematics.

When did I claim no one at SI held your views on anthropics? And I really don't think anthropics could be called straightforward by anyone.

There are pathological, intuitively confusing cases such as the number example; reading a number is incredibly selective for it being that number, so the update, in fact, does pull the probability up.

Congratulations, you understood the point. Similarly, decent arguments are highly diagnostic of philosophical ability because most people couldn't make an even half-assed argument if they sat down and spent all day at it; by LW standards, most philosophy grads can't find their asses either, and that's a very selective filter as well (philosophy majors are the highest-scoring group on the GRE for both verbal and writing sections, and are around 4 or 5 for the math section below physics & mathematics as one would expect). Making an argument that doesn't instantly fail is sadly so rare that just seeing one moves you a long way towards '1 in a million'.

Yudkowsky is not exactly Ayn Rand level popular, is he? If that's what you're after, pick anyone more famous than Yudkowsky and you're done. Easy.

I never said that fame scaled smoothly with importance. If I had to put the cutoff where fame stops adding additional evidence, I think I'd put it somewhere upwards of a Wikipedia article.

Look back up, Muehlhauser has stated that it is a tough task choosing someone of Yudkowsky's level of philosophical ability.

Sure. You're fishing from a limited pool to begin with: there aren't many professional philosophers these days, their numbers are probably shrinking as humanities programs get pressured. To put some numbers in perspective: the annual East coast meeting of the American Philosophical Association (APA) is the biggest single gathering of philosophers (tenured professors, associates, adjuncts, grad students, etc) in the world as far as I know. It numbers ~2000 attendants. Making things even more difficult, if I were one of them, I doubt I would spend much time on MIRI/FHI-related issues even if I were a true believer: it'd be way too risky for my already extremely precarious career. (Recruiting-wise, it might be best just to try to find computer science people and have them try their hand at philosophy; there's a lot of them, they're almost as smart in general, they have direct familiarity with a lot of the issues, they'll have the right intuitions about things like 'computers really are just machines that do what the programs say', and funding is a lot easier for them.)

By the way there's a pattern, various Ayn Rands and Keith Ranieres and Ron Hubbards and other self improvement gurus slash philosophers slash world saviours are popular philosophers among non-philosophers but not recognized by other philosophers.

I've actually never heard of Keith Raniere despite growing up in NY and visiting RPI; Wikipedia doesn't do a good job of describing what's so bad about it... ("Expensive brainwashing"? Brainwashing doesn't work, that's why cults have annual attrition rates in the double-digits.)

Anyway; yes, I would agree that the previous points also increase the chance EY would fall into that category of frauds. After all, such frauds are also pretty rare, so it's hardly impossible for evidence to increase our beliefs both that EY is a good philosopher and also such a fraud.]

(An example: houses catching on fire are rare. Houses not on fire with red spotlights around them are also rare. If I see in the sky above a house in the woods flickering red light, this is consistent with both the house being on fire and them having set up spotlights for a party; and my beliefs in the possibility of a fire and the possibility of spot spotlights will both increase quite a bit even though they're mutually exclusive scenarios.)

What evidence gave you this impression?

Yes. But he is a barely mediocre philosopher who is in no position to recognise real talent, whether EY's, if it exists, or anyone else's. He confuses ability with style or adherence to doctrines that he approves of.

How much of CSA have you read? Search for the sweet-spot just before Luke discovered LW and you should find high level philosophy going on.

Whatever caused your slide into jadedness?

by LW standards, most philosophy grads can't find their asses

By philosophy standards, most LWers can't find their arses.

By arse standards, most philosophy grads can't find LW.

(Sorry, what was this permutation meant to accomplish?)

Do you mean to claim that people who are competent by the standards of philosophy would not treat other people like this?

If so, what is your estimate of the percentage of people with graduate degrees in philosophy who are competent by the standards of philosophy?

anyone competent by any standard would not do that by accident

So, I know lots of people who are demonstrably competent by the standard of "computer programmer," just to pick an example, who would absolutely do stupid shit like that without intending to be hurtful. So if you really mean "by any standard," then I simply don't believe you.

Conversely, if you just meant that as rhetorical exaggeration, I'm still left with the question of what standards you're actually talking about. Which is what I was trying to clarify with my questions, which you ignored.

Not sure how to proceed from here. Probably best to drop it.

I choose the profession as my example because I know a lot more computer programmers than any other single profession.
Almost all the computer programmers I know are not self-obsessed jerks.
I'll answer your questions when you answer mine.

Says the person whose whole argument of opposition to compatibilism was basically the cry "but where is the choice?!?"

There are always higher levels. If nothing else, you can invent them yourself.

That's what came to mind after reading this post, after reflexively comparing how intelligent I think I am to how intelligent I perceive the author and commenters to be.

Another thing that came to mind was a grumpy sense that the whole issue had not been framed in a useful way, and an urge to meddle with how the ideas are arranged.

When I see someone's work who is at higher levels than my own current understanding and abilities allow me to achieve, (notice I am not phrasing that as if it is innate and cannot be learned,) I often feel frustrated at the poor quality of the educational materials or culturally transmitted knowledge that I have come across so far, the lack of map to get there.

I also usually feel relief, and, depending how much I care about the ability and subject matter, sometimes fascination at the work itself, and at the puzzle of how to get closer to that level of ability. I also feel more hope for the world.

It is easier to see brilliance than create brilliance.

In some fields of study there are many people working on a problem. But if you are the only one who you know of who devotes a lot of time to the problem, who cares that much, and who is taking an approach to the problem that is not clearly incorrect, then, it can get...lonely. Except lonely is not the right word. It is not like being lonely for social reasons. It is the murkiness of familiar uncertainty, of having no reliable measuring tools, of staying there for a long time. At times, it can be disorienting.

I'm not sure I am actually addressing the topics in the post, because the post seems to be a mix of several ideas, and the way it is written feels incomplete and imprecise, at least in so much as it conveys meaning to me. And, it is a ~5 year old post, probably the people involved have updated their thinking by now. Nevertheless, it struck a chord with me.

If there was gasp someone who was smarter than you in the ways you care about being smart yourself...then so what? I'm not sure what that has to do with your ability to revolutionize your field, or the sense of being limited by past choices, or concerns you are not using your time in the optimal way, or the way in which expertise and levelling up alters one's perceptions of others and their work. Perhaps these issues all connect in some way, but they seem like they have been unnecessarily conflated into a bit of a mess, to me.

If anything, I would think that someone else being able to achieve something great would make it more likely that you could do so, by showing that it is humanly possible, by giving an example to study and learn from, by expanding the knowledge available to humanity. Having more building blocks with which to build new stuff is, usually, a net benefit.

Also, some things get invented all at once, others in incremental stages. The history of the electric guitar, how several people tinkering with technology led to the instrument we have today, is a counterexample to the single brilliant person model of discovery and invention.

If someone comes along and solves all the problems you've been working on, wouldn't that be a good thing? Do you really not have confidence in your ability to take the next step, or to find something else to work on? I personally have a long list of things that I want to do, and to do them all would take several lifetimes, if lifespans remain about the same as they currently are. Wouldn't that free you up to do something else? I do not get the sense that you have any shortage of things you desire to spend your time on or accomplish.

If no one solves it, if you do not finish, or even fruitfully work on your chosen task, if you fail, is that the bigger problem? To put in so much time and effort, and have it be for nothing? I don't have much to say to that one, except that this is the risk of working on interesting important problems.

Can anyone tell me whether Jaynes' book can be read and understood without any particular formal training? I do know the basic concepts of probability, and I usually score around the 85th percentile on math tests... And how hard/time-consuming exactly will the book be? I am employed in a somewhat high pressure job on a full time basis...

You'll note that I don't try to modestly say anything like, "Well, I may not be as brilliant as Jaynes or Conway, but that doesn't mean I can't do important things in my chosen field."

Because I do know... that's not how it works.

Maybe not in your field, but that is how it usually works, isn't it?

(the rest of this comment is basically an explanation of comparative advantage)

Anybody can take the load off of someone smarter, by doing the easiest tasks that have been taking their time.

As a most obvious example, a brilliant scientist's secretary. Another example: a brilliant statistician that employs a programmer, who turns his statistical and computational ideas into efficient, easy-to-use software. He doesn't have to be the best programmer, and doesn't have to be that great at statistics, but he allows the statistician to publish usable implementations of his statistical methods without having to code them himself.

Or, here's another one: I've heard MIRI needs a science writer, or needs funding for one. You don't have to be Eliezer Yudkowsky-level at thinking about FAI to save Yudkowsky the time it takes to write manuscripts that can be published in science journals, and then Yudkowsky uses that time for research.

This is "important work." It's not the kind of important work Jaynes or Conway does, and it doesn't put your name in the history books, and if that's what was meant by the article I have no disagreement. But by any utilitarian standard of importance, it's important.

For whatever reason, the sense I get of Jaynes is one of terrifying swift perfection—something that would arrive at the correct answer by the shortest possible route, tearing all surrounding mistakes to shreds in the same motion. Of course, when you write a book, you get a chance to show only your best side. But still.

Just reminded me of a Lord Acton's quotes : "Judge character at its worst, but talent at its best." (Paraphrased from memory)

The 'thousand years old' vampire impression could be close to truth. My understanding is that people like Jaynes think very long hours, and can clock as much relevant brain time by their thirties as a person of lesser mental endurance may clock in centuries. And it is entirely possible that Jaynes did as much math by the time he wrote the book (correct math - checked and verified) as a hobbyist would in thousands years.

Contrarian view about Jaynes' super-smartness,- from David Chapman.

Apparently, Jaynes "was completely confused about the relationship between probability theory and logic." and "There’s strong evidence that when people tried to de-confuse him, he pig-headedly refused to listen."

Honestly, my favorite thread I've read so far. I lived a similar scenario so many time, and while I doubt any of my "level above mine models" are anywhere near Jeynes, I'm very proud knowing I did manage to catch up and even surepress some. In some parts, thanks to 'less wrong' and 'Rationality from AI to Zombie'.

So thank you (Eliezer and many others on this blog) for sharing your experience and knowledge. You're some of my best teachers

Oh. 

All this time, and I've never thought once that Eliezer could be thinking about other people nearly the exact way I thought about him.