The limiting factor is organizational. Scientific activity can scale; recognition or propagation of it doesn't.

While failure to recognize & propagate new scientific discoveries probably explains some of our apparent deficit of current scientific geniuses, I think a bigger factor is just that earlier scientists ate the low-hanging fruit.

(I have no idea whether a similar effect would kick in for superintelligences and throttle them.)

This seems an important issue to me.

Back up. The population of Europe was under 200 million in 1700, less than a sixth of what it is today. The number of intellectuals was a tiny fraction of the number it is today. And the number of intellectuals in Athens in the 4th century BC was probably a few hundred. Yet we had Newton and Aristotle.

Those places were selected for having Newton and Aristotle though.

The limiting factor is organizational. Scientific activity can scale; recognition or propagation of it doesn't.

What leads you to be confident that these are the bottlenecks?

I measured science and technology output per scientist using four different lists of significant advances, and found that significant advances per scientist declined by 3 to 4 orders of magnitude from 1800 to 2000.

Interesting. Is your research up online?

On the other hand, merely recognizing and solving the organizational problems of science that we currently have would produce results similar to a fast singularity.

You mean, we would have a lot more effective research, quickly? Or something more specific?

yet we do not have 1000 Mozarts or 1000 Beethovens

What do you mean by this? We have plenty of composers and musicians today, and I'd bet that many modern prodigies can do the same kinds of technical tricks that Mozart could at a young age.

I measured science and technology output per scientist using four different lists of significant advances, and found that significant advances per scientist declined by 3 to 4 orders of magnitude from 1800 to 2000. During that time, the number of scientific journals has increased by 3 to 4 orders of magnitude, and a reasonable guess is that so did the number of scientists.

I'd be really interested in reading more about this.

Yet we had Newton and Aristotle.

Compare the complexity of F=MA to string theory. The difficulty of science is going up by orders of magnitude as the low hanging fruit are eaten.

yet we do not have 1000 Mozarts or 1000 Beethovens.

We have over 1000 genres of music. Sure, not everyone can be recognised as the best musician of a generation by definition, but I think we could arguably be producing 1000 times more good music than at the time of Mozart.

Good post.

First of all, knowledge is partially ordered. A bunch of lesser-known results were required before Einstein could bring together the mathematical tools and physics knowledge sufficient to create relativity. True enough, this finding may have come much later, if not for Einstein, but dozens of others built predecessor results that also required great insight.

Similarly, we should not decry the thousands of biologists who have been cataloging every single protein, its post-translational modifications and its protein-protein interactions in exhaustive detail. Some of this work requires a great deal of cleverness each time.

A portion of the phenomenon you are talking about can be addressed by referencing Kuhn: We have periods of normal science, with many people giving input, and a building tension where twenty pieces fall into place and (frequently interdisciplinary) thinkers visit a problem for the first time.

In other cases the critical breakthrough has to be facilitated using new tools that generate new breakthroughs. When these tools require advances in component technology, you have a large number of engineers, testers and line workers feeding their talents into discoveries for which only a few get credit sometimes.

If these components require days or months of "burn-in" testing to judge their reliability, a superintelligence might have limited advantage over people in reducing the timeline.

Sometimes discovery relies on strings of experiments which by their nature require time and cannot be simulated. Our current knowledge of human biology requires that we follow patients for many years before we know all of the outcomes from a drug treatment.

Initially, at least, a superintelligent drug developer would still have to wait and see what happens when people are dosed the drug over the course of many years.

If a cosmic event can only be observed once in a decade, a superintelligence would not have the data any sooner, short of inventing some faster-than-light physics we do not have today.

Present-day humanity is a collective intelligence that is clearly 'superintelligent' relative to individual humans; yet Bostrom expresses little to no interest in this power disparity, and he clearly doesn't think his book is about the 2014 human race.

So I think his definitions of 'superintelligence' are rough, and Bostrom is primarily interested in the invincible inhuman singleton scenario: the possibility of humans building something other than humanity itself that can vastly outperform the entire human race in arbitrary tasks. He's also mainly interested in sudden, short-term singletons (the prototype being seed AI). Things like AGI and ems mainly interest him because they might produce an invincible singleton of that sort.

Wal-Mart and South Korea have a lot more generality and optimization power than any living human, but they're not likely to become invincibly superior to rival collectives anytime soon, in the manner of a paperclipper, and they're also unlikely to explosively self-improve. That matters more to Bostrom than whether they technically get defined as 'superintelligences'. I get the impression Bostrom ignores that kind of optimizer more because it doesn't fit his prototype, and because the short-term risks and benefits prima facie seem much smaller, than because of any detailed analysis of the long-term effects of power-acquiring networks.

It's important (from Bostrom's perspective) that the invincible singleton scenario is defined relative to humans at the time it's invented; if we build an AGI in 2100 that's superintelligent relative to 2014 humans, but stupid relative to 2100 humans, then Bostrom doesn't particularly care (unless that technology might lead to an AI that's superintelligent relative to its contemporaries).

It's also important for invincible singleton, at least in terms of selecting a prototype case, that it's some optimizer extrinsic to humanity (or, in the case of ems and biologically super-enhanced humans -- which I get the impression are edge cases in Bostrom's conceptual scheme -- the optimizer is at least extrinsic to some privileged subset of humanity). That's why it's outside the scope of the book Superintelligence to devote a lot of time to the risks of mundane totalitarianism, the promise of a world government, or the general class of cases where humanity just keeps gradually improving in intelligence but without any (intragenerational) conflicts or values clashes. Even though it's hard to define 'superintelligence' in a way that excludes governments, corporations, humanity-as-a-whole, etc.

(I get the vague feeling in Superintelligence that Bostrom finds 'merely human' collective superintelligence relatively boring, except in so far as it affects the likely invincible inhuman singleton scenarios. It's not obvious to me that Hansonian em-world scenarios deserve multiple chapters while 'Networks and organizations' deserve a fairly dismissive page-and-a-half mention; but if you're interested in invincible singletons extrinsic to humanity, and especially in near-term AI pathways to such, it makes sense to see ems as more strategically relevant.)

Bostrom's secondary interest is the effects of enhancing humans' / machines' / institutions' general problem-solving abilities relative to ~2014 levels. So he does discuss things other than invincible singletons, and he does care about how human intelligence will change relative to today (much more so than he cares about superintelligence relative to, say, 900 BC). But I don't think this is the main focus.

WM raises issues of computational complexity which have so far been ignored. If working memory is the set of concepts that are currently being matched against each other, then the complexity of the matching is probably n^2, If it is the set of concepts all permutations of which are being matched against variables in rules, the complexity is n!. It's easy to imagine cognitive architectures in which the computational capacity needed to handle 9 items in WM would be orders of magnitude higher than that needed to handle 5. I suspect that's why our WM is so limited, particularly in light of the fact that WM appears to be highly correlated with intelligence (according to Michael Vassar).

To me, this seems like an issue with the definition of 'chunk of information'. Sure, maybe I can only remember a few at a time, but each chunk has a whole bunch of associations and connected information that I can access fairly easily. As such, my guess is that these chunks actually store a very large number of bits, and that's why you can't fit too many of them into short-term memory at once. Of course, you could do even better with better hardware etc., but this seems to just be an instance of the point "humans have finite memory, for every finite number there's a bigger number, therefore we could make machines with more memory".

To the extent that working memory is just like having a sheet of paper to one side where you can write things, we more or less already have that, though I agree it could be better integrated.

And faster, by many orders of magnitude! Modern PCs already have ram capacities on the order of tens of gigabytes. If we're talking about simply written words (as opposed to arbitrary drawings, the storage space requirements for which are somewhat trickier), that's not the equivalent of a page, or a book - it's on the order of a library. And they can read or overwrite the whole thing within a few seconds.

Yes, in principle you can do the same thing by hand. In practice, writing out even one full ram dump by hand would probably take longer than a current human lifetime.

That by itself would provide a speed-type superintelligence advantage, to the extent that flat (i.e. not associative to the same extent human memory is) memory is a limitation on our intelligence.

Concretely, we could imagine a model in which the brain automatically explored all ways of composing two concepts in working memory (as a kind of automatic architectural feature), or even did something more elaborate (e.g. explored all possible subsets). In this scenario, it would be very expensive to scale up the size of working memory while retaining the same characteristics, though it wouldn't be an in principle obstruction.

Haven't read the book, so forgive me if this is covered.

It is easier to imagine this as the laws of physics being 10 times slower, then multiply everything by 10.

[begin slow-physics POV]

Manufacturing and agriculture slow significantly. Everything weighs a tenth of its current weight, but muscles and engines are 10 times weaker. Inertia and viscosity become important. The marginal cost of per km of shipping goes up by about a factor of 10, but the fixed costs of shipping remain about the same. With the fall in trade, specialization decreases. Solar radiation is 10 times weaker, and plants grow 10 times slower. Renewable energy sources become 1/10th as appealing (non-renewables are better). Smelting, tempering, and other time-bound processes proceed at 1/10th the current rate. Capital intensive manufacturing steps become 10 times as capital intensive. Real wages fall.

The service sector (which is a majority of GDP) is not nearly as slowed. Since the day is now 240 hours, sleep cycles and work schedules are no longer connected with the sun. The whole planet can coordinate on a work schedule, maybe with 8 30-hour sleep cycles per earth rotation. Computers are as fast as in mid 2000s. Email and other telecommunication is virtually unaffected (voice data increases by 10x, but voice data is small these days). People type slower, but we can easily invent less laborious keyboards. Commutes are difficult, as cars race on highways at 10 km/h but are only as maneuverable as cars moving at 100 km/h in our world. People move closer their offices and telecommuting becomes much more commonplace.

All in all, it seems like a slowdown in physics would cause a slowdown in our economy, but we would rapidly adapt.

[end slow-physics POV]

The rapid change in comparative advantages would immediately expose much low-hanging fruit (moving to more labor-intensive manufacturing techniques, for example). Growth would increase rapidly as industry re-optimizes for the new high-speed labor force. Growth slows after this low-hanging fruit is plucked.

On this subject, I'd like to link Stanovich on intelligence amplification.

Difficult question. Do you mean also ten times faster to burn out? 10x more time to rest? Or due to simulation not rest, just reboot?

Or permanently reboot to drug boosted level of brain emulation on ten times quicker substrate? (I am afraid of drugged society here)

And I am also afraid that ten time quicker farmer could not have ten time summer per year. :) So economic growth could be limited by some botlenecks. Probably not much faster.

What about ten time faster philosophic growth?

Breaking this down a bit:

-Ten times faster does not help people driving vehicles that much, unless they can use the time to multi-task.

-For people who carry things around to complete their job, or manipulate objects, we have to decide whether they are physically able to do those actions faster, or are they just able to think faster.

Presumably, we're holding physical running, walking and carrying speeds the same, and people are thinking a lot faster. Thus, people can plan their farming enterprise much more effectively, but they still need a lot of people to actually pick the crops.

In this scenario, we would end up with a lot of people who have an expert-level mastery of many professions. Since it does not take much time to learn anymore, a lot of people would be MD, JD, MBA, triple PhDs, know how to operate dozens of pieces of equipment, know ten languages and be able to design and build their own house or office.

Apparently, all of that happens for some people before their 10th birthday-but what did this speed-up mean for their emotional development? Do they start working at 5 years old? Do we end up with very expert people who still cannot manage their temperament, thereby making them very effective threats?

One good question is the degree to which speed translates into quality-how many more people would be able to write a coherent 100-page document with this new ability? Not clear.

Some would use all of the additional time to play 10x more video games, watch 10x more soap operas or to delve deeper into Sufi mysticism. A segment of the population would use these gifts well, others would not.

Team leaders frequently would do much more of the work of their projects themselves, since after their interdisciplinary education they would not require specialists for many task. However, when necessary, these leaders could have fifty people reporting directly to them, rather than just 5.

Ten times faster communication between people would allow them to correct many social mistakes prior to damage, to make very detailed business contracts, and sometimes to avoid armed conflict. Critical relationships where both parties have an incentive toward preserving the relationship would improve.

There would be more time to make sure that you were buying the right product, and more time to make sure that your sales pitch was as effective as possible. However, sometimes that would only mean 20% more sales resulting in a 15% better product purchased.

The queues in store check-out lines evaporate, and wait times on phone interactions with businesses and governments also fall. An array of telepresent services becomes possible.

At the same time, however, our ability to search for new relationships would also improve.

In the event of business competition, some strategic advantages would multiply, others would be neutralized. The business who can make purchases for 3% less has a much greater advantage and defeats equal competition much more quickly.

Identifying your adversaries before they identify you would become extremely important, because you could set up a multi-step plan to defeat them quite quickly, or serve ten people with just a slightly better product where you otherwise would've only gotten one.

Second, as time goes on, we have to try harder and harder not to see the answers to the "eternal problems" lying in front of our noses, because we're still hoping to find different answers.

What would explain all the questions to which we are unwilling to accept the answers falling in the domain of philosophy? Or are these merely the ones where we are not forced yet to accept them?

Eliezer Yudkowsky has written about the idea of a "codic cortex"; that is, a specialized mental module for modelling the behavior of executable code.

And something like that would be really useful! For instance, there's fundamentally no good reason to have any implementation bugs when writing code, or to not easily notice them when reading it. The techniques for how to prove code correctness are well known; but in practice, for a human programmer to actually use them is so expensive (in terms of productivity) that it's usually more efficient to skip them, and find and fix bugs after the fact.

This is despite the fact that humans already have (in absolute terms) very good models of how the code they write works; if we didn't, we couldn't do nontrivial programming at all. But those models are sloppy, and there's some details that are easy to miss for humans. If we built the models instead from a formal and precise analysis of the code on hand, we'd get much better predictions out of them.

A lot of programming language/environment development has been concerned with having the compiler or runtime handle certain things (translating high-level structures to machine code, garbage collection, type safety, etc.) so that the programmer doesn't need to worry about getting them right - both so they don't end up getting them wrong, and so that having to painstakingly get them right doesn't drain their productivity. But such approaches usually come with some performance cost, and in the end they're crutches to deal with the fact that humans are no good at programming. None of them would be necessary for an intelligence that had a decent specialized module to handle code modelling.

Autonomous systems are advancing toward being able to create high-precision, 3d renderings of indoor or outdoor locations in milliseconds, then recognize the objects in the location. This technology has room to improve rapidly, and we may suddenly find that it dramatically exceeds our own abilities without a corresponding level of progress toward what people are calling AGI.

If you continuously improve a system's speed, then the speed with which each fixed task can be accomplished will be continuously reduced. However, if you continuously improve a system's quality, then you may see discontinuous jumps in the time required to accomplish certain tasks. So if we think about these dimensions as possible improvements rather than types of superintelligence, it seems there is a distinction.

This is something which we see often. For example, I might improve an approximation algorithm by speeding it up, or by improving its approximation ratio (and in practice we see both kinds of improvements, at least in theory). In the former case, every problem gets 10% faster with each 10% improvement. In the latter case, there are certain problems (such as "find a cut in this graph which is within 15% of the maximal possible size") for which the running time jumps discontinuously overnight.

You see a similar tradeoff in machine learning, where some changes improve the quality of solution you can achieve (e.g. reducing the classification error) and others let you achieve similar quality solutions faster.

This seems like a really important distinction from the perspective of evaluating the plausibility of a fast takeoff. One quesiton I'd love to see more work on is exactly what is going on in normal machine learning progress. In particular, to what extent are we really seeing quality improvements, vs. speed improvements + an unwillingness to do fine-tuning for really expensive algorithms? The latter model is consistent with my knowledge of the field, but has very different implications for forecasts.

Turing machine (very slowly), which we know/suspect can do everything computable. It would seem then that a quality superintelligence is just radically faster than a human at these problems.

I think that statement is misleading, here. To solve a real-world problem on a TM, you do need to figure out an algorithm that solves your problem. If a Dark Lord showed up and handed me a (let's say ridiculously fast compared to any computer realizable on what looks like our physics) UTM - and I then gave that UTM to a monkey - the monkey may have a fairly good idea of what it'd want (unlimited bananas! unlimited high-desirability sex partners!), but it wouldn't have any idea of how to use the UTM to get it.

If I tried to use that UTM myself, my chances would probably be better - I can think of some interesting and fairly safe uses to put a powerful computer to - but it still wouldn't easily allow me to change everything I'd want changed in this world, or even give me an easy way to come up with a really good strategy to doing so. In the end, my mental limits on how to decide on algorithms to deal with specific real-world issues would still be very relevant.

Humans cannot simulate a Turing machine because they are too inaccurate.

There's interesting work on how culture accumulates in function of distinct kinds of imitation. Richerson and Boyd have done much of the theorizing. Tim Tyler who frequently publishes on Lesswrong wrote a book on Memetics which might give some hints. Finally Alex Mesoudi has specifically studies levels of optimality in different copying strategies.

Define intelligence how you like, there are a few capabilities not in this list which will significantly influence the capabilities of autonomous systems, including how beneficial, threatening or capable they are:

Sensory systems:

Both machines, and future collective intelligences of people and machines, are benefiting from an exponential explosion the ability to sense. Everything is improving at a vast pace: telescopes, microscopes, detection in every wavelength, sound, chemistry, DNA/RNA sequencing, mass spectrometry, surveillance in thousands of locations, internet of things, the list goes on and on.

If we want to understand the future of intelligence, we have to incorporate the explosion of sensory input into our thinking.

I am not ready to claim that spirituality is irrelevant to the discussion. The difference between factual knowledge, calculation speed and "wisdom" does seem relevant, as others have pointed out earlier in the thread.

However, we'll have to re-frame questions about spiritual issues in some way to bring it in...

Suppose one aspires to become a Bodhisatva-a being who is capable of entering into an unfettered celestial existence, but who instead stays behind to help other people and sentient beings find their way. (If we are a bit less ambitious, perhaps we might, depending on our tradition, aspire to become a tzadik, a marja, or saintly but not a supernatural savior).

Among other things, the Bodhisatvas have removed themselves from ego and forms of physical desire. They are spiritually superior and ready to move outside of the cycle of reincarnation as lower beings. Yet, when they take the form of a human, they would eat as an instrumental goal toward accomplishing their end.

The Bodhisatva's kind of spiritual superiority does seem to differ from what Bostrom and the rest of us call "Superintelligence." It may, however, relate in some ways...

Just a thought experiment...

For example, a peculiar version of a Bodhisatva might be said to have a utility function-in fact a marvelous, selfless utility function. Also, it will succeed at bringing more people out of the darkness if it engages in recursive self-improvement of its capabilities.

One aspires to become a Buddha or a Bodhisatva, and the claim is made by some that they, others who they know, or historical figures have reached this condition. For the most part, however, this goal is aspirational in nature and works to improve people's behavior toward one another, and toward other sentient beings, in this life.

Based on what we now know about neurophysiology, the human brain is going to have a lot of trouble reaching any form of true enlightenment on its own. Emotion and desire re-surface even in the most virtuous or contemplative among us. Seemingly, we are never going to be truly saintly for the rest of our lives, although perhaps we can manage to have some pretty virtuous days interspersed among our failures.

I am not ready to make a proposal, but if enlightenment or spiritual purification is our goal, we might have to resort to some kind of augmentation to make it to the next step in that direction. Perhaps a lot of meditation or ethical contemplation is one approach, perhaps we may think of others...

Strictly an augmentation of intelligence, however, is not necessarily going to bring about this outcome.

The ability for autonomous systems to manipulate objects in the real world may relate to other forms of intelligence, but understanding future threats and benefits requires that we break this set of skills out and consider it separately.

We already live in an age of autonomous vehicles. To understand the future, we have to forecast how this revolution will play in with other advances in intelligence.

The forward progress of robotics is different from the intelligences in this chapter. Robotics may benefit from AI planning and object recognition, but robots are hardware/software solutions.

Superintelligence will seem a lot more "super" if fundamental issues relating to object recognition, navigation indoors and object manipulation are solved first.

The analysis improves if we separate out and take a closer look at the differences between memory and reasoning.

A critical question with Neurons is how to account for the amount of internal state they contain.

A cell can be in a huge number of internal states. Simulating a single cell in a satisfactory way will be impossible for many years. What portion of this detail matters to cognition, however? If we have to consider every time a gene is expressed or protein gets phosphorylated as an information processing event, an awful lot of data processing is going on within neurons, and very quickly.

It appears that vastly simplifying all of this detail in simulation may work out pretty well-but there is a big argument between Markram and IBM's neuromorphic people about this issue.

We really need to delve deep on this and get all of the latest thinking in one place.

The first part of the sentence compares brains with current computers, the second part with theoretical possible artificial minds. Modern computers are less efficient than brains in pretty much every other respect.

Wisdom is goal-oriented thinking. Intelligence is the type of calculations that should go into optimization towards goals but are often misdirected. You can optimize steadily but weakly towards a goal, or you can make calculations and make decisions with high power, but shooting in all directions.

Neither Bostrum nor our discussion properly distinguishes between fluid and crystallized intelligence.

Fluid intelligence is the capacity to think logically independent of aquired knowledge.
Crystallized intelligence is the ability to use skills, knowledge and experience.

Howard Gardners concept of multiple intelligences is focussing on aquiring crystallized intelligence:

As the name [multiple intelligences] indicates, I believe that human cognitive competence is better described in terms of a set of abilities, talents or mental skills, which I call intelligences. [...] Such a theory has important educational implications.

(Gardner 2006 Multiple Intelligences - New Horizons p6)

The word wisdom implicates experience and higher age. Wisdom does not need to be universal knowledge. Years of experiences in some restricted domains together with high inter- and intrapersonal intelligence build wisdom.

For our discussion on engineering seed AIs it is important to know that this AI needs wisdom in engineering. This is: high fluid intelligence, high creativity (a concept different from intelligence), and a lot of practical experience and theoretical knowledge about hard- and software engineering.

Mea Culpa for falling behind.

Bostrom mentions ( p55 ) that the increase in human population and organizational complexity since the Pleistocene suggests that we have created "superintelligence relative to a Pleistocene baseline". And we have radically changed the world, in an accelerating fashion, from that baseline. Yet it seems plausible that Pleistocene tribes could have "slipped through the cracks", perhaps in the Amazon or Pappa New Guinea, and continued living much the same style of existance since then. Since this seems less plausible for what we conventionally refer to as superintelligence, this suggests there is some sort of dis-analogy that is being overlooked here.

His point about organizations being good at parallel, independantly verifiable tasks was interesting. It seems that much of human progress has consisted of

• A very clever person working out how to do something once.
• Another clever person working out how to break it down into 1000 steps that can be outsourced to ordinary people, enabling mass production at reasonable cost.

now we are beginning to replace the second step with

• Another clever person works out how to code it up, enabling mass production at very low cost

but perhaps AGI will make the second step redundant. This would not only reduce "time to market", but also allow a whole new class of innovations to be realized on a significant scale.

A higher quality intelligence than us might, among other things, use better heuristics and more difficult analytical concepts than we can, recognize more complex relationships than we can, evaluate its expected utility in a more consistent and unbiased manner than we can, envision more deeply nested plans and contingencies than we can, possess more control over the manner in which it thinks than we can, and so on.

A more general intelligence than us might simply have more hardware dedicated to general computation, regardless of what it does with that general ability.

In practice for "quality" we normally have a yardstick for performance which is being improved continuously (e.g. success probability, quality of a solution to an optimization problem, ability to win in a game), while for generality there is often no such yardstick. At best this seems like a difference of degrees rather than a difference in kind though.

I don't know if there is some more convincing distinction. I can't think of any arguments that depend on a distinction.

Perhaps a more general intelligence can do well in a wider variety of circumstances, whereas a higher quality intelligence can do better in those circumstances, by seeing better solutions etc rather than just being faster.

Intelligence is the ability to achieve complex goals in complex environments.

The greater the total complexity of the set of goals that the organism can achieve, the more intelligent it is.

Goertzel 2006, p43

Bostrum unsatisfactorily defines quality superintelligence in a self referencing circle by vastly qualitatively smarter (p56). It would have been better to name the ability to solve problems of vastly higher complexity.

Intelligence of higher generality covers more domains.

You can improve in intelligence by generalizing ('My intelligence improved in generality'), or by further investing in what you're good at ('My intelligence improved without improving in generality'). It seems like we could mean two different things by 'generalizing'.

Suppose four skills exist, A,B,C,D; and my skill level can either be low (0), mediocre (1), high (2), or very high (3). If I start off with A=0, B=1, C=2, D=2, then 'generalizing' might mean improving A or B more than I improve C or D. Alternatively, 'generalizing' might mean improving in more skills, rather than in just one. On the former conception, 'raise A to 2' increases my intelligence's generality more than 'raise C to 3 and D to 3'; on the latter conception, the reverse is true. There's plug-the-gaps generalization, where you try to get rid of your weak points; but there's also spread-the-love generalization, where you try to find self-improvements that will impact your problem-solving ability in as diverse a range of problems as possible.

'Qualitative intelligence improvements' seems like a grab-bag for 'all the kinds of intelligence improvements that we don't usually measure in any simple and direct way'. We routinely talk about, e.g., the speed, number, and computing power of computers, in terms of simple numerical values; we don't routinely do the same for computers' language-processing abilities, so that goes in the 'qualitative' bag, at least for the moment. Improving in qualitative intelligence could take almost any form; it seems like a less natural category than 'generality'.

Katja mentioned some sources to look into. Also the field of Evolutionary Psychology as well as Textbooks on Cognitive Neuroscience will be divided by different kinds of intelligence that seem (to the psychologists and neuroscientists studying them) distinct. Goertzel has interesting perspectives.

The Superorganism is humanity's best tackle so far at understanding collective intelligence made out of pretty dumb little creatures (ants, bees, wasps). It should be given more value for it's insights than it currently is within the LW community.

Once thing going on here is that later discussion will be focused on recursively self-improving autonomous systems. We would like to know, for instance, whether and when software will be able to program other useful software.

I am not ready to claim that spirituality is irrelevant to the discussion. The difference between factual knowledge, calculation speed and "wisdom" does seem relevant, as others have pointed out earlier in the thread.

However, we'll have to re-frame questions about spiritual issues in some way to bring it in...

Suppose one aspires to become a Bodhisatva-a being who is capable of entering into an unfettered celestial existence, but who instead stays behind to help other people and sentient beings find their way. (If we are a bit less ambitious, perhaps we might, depending on our tradition, aspire to become a tzadik, a marja, or saintly but not a supernatural savior).

Among other things, the Bodhisatvas have removed themselves from ego and forms of physical desire. They are spiritually superior and ready to move outside of the cycle of reincarnation as lower beings. Yet, when they take the form of a human, they would eat as an instrumental goal toward accomplishing their end.

The Bodhisatva's kind of spiritual superiority does seem to differ from what Bostrom and the rest of us call "Superintelligence." It may, however, relate in some ways...

Just a thought experiment...

For example, a peculiar version of a Bodhisatva might be said to have a utility function-in fact a marvelous, selfless utility function. Also, it will succeed at bringing more people out of the darkness if it engages in recursive self-improvement of its capabilities.

One aspires to become a Buddha or a Bodhisatva, and the claim is made by some that they, others who they know, or historical figures have reached this condition. For the most part, however, this goal is aspirational in nature and works to improve people's behavior toward one another, and toward other sentient beings, in this life.

Based on what we now know about neurophysiology, the human brain is going to have a lot of trouble reaching any form of true enlightenment on its own. Emotion and desire re-surface even in the most virtuous or contemplative among us. Seemingly, we are never going to be truly saintly for the rest of our lives, although perhaps we can manage to have some pretty virtuous days interspersed among our failures.

I am not ready to make a proposal, but if enlightenment or spiritual purification is our goal, we might have to resort to some kind of augmentation to make it to the next step in that direction. Perhaps a lot of meditation or ethical contemplation is one approach, perhaps we may think of others...

Strictly an augmentation of intelligence, however, is not necessarily going to bring about this outcome.