Report from a civilizational observer on Earth

post by owencb · 2022-07-09T17:26:09.223Z · LW · GW · 12 comments

Contents

  Intellectual automation
  The Omohundro-completeness threshold
  Superhuman generalization
  Anthropological note
None
12 comments

[translated into English by the author]

I have arrived in [2022]. There is one species, humans, that is building civilization. (A few other species exhibit potential but have not crossed the threshold of runaway economic development or cultural accumulation; it is now unlikely that they ever will unless a deliberate choice is made to allow them to do so.)

There are around 12^9.2 humans. They are organized into loose and overlapping political structures, including firms, [countries with systems of law one cannot opt out of], and [religions and hobby groups]. There is a degree of [law, treaties, and custom] constraining these groups, but there is no strong political unification, and sometimes there is significant [violence causing destruction of persons or property].

Most of the physical activity of civilization is done by machines operated by humans, although a significant minority is still done directly by humans, and a significant-and-increasing minority is done by machines operated by [rule-governed thinking-machines].

In some domains a large majority of the intellectual labour of civilization is done by thinking-machines, but these tend to be [rule-governed thinking-machines] following precise instructions laid down by humans. Most key intellectual work is still done by humans (although a lot of the communication of this is mediated by [rule-governed thinking-machines]).

In the last econoyear^[1], civilization has built nontrivial [patterning thinking-machines]. And in the last economonth^[2], these have been applied to human language with significant success. [Patterning thinking-machines] are now capable of doing sophisticated tasks that humans might previously have been paid to do. And since the language they learned encoded a lot of civilizational knowledge, these [patterning-thinking-machines] now implicitly contain a large fraction of civilizational knowledge (at least at a [shallow/indexing] level).

 

Intellectual automation

The next several economonths will likely represent a period of acceleration for civilization. It is unclear whether they will contain an emergent [post-civilization], though it is most likely that that will occur within the next 24 econoyears, and maybe much sooner.  It is currently unclear what the nature of any emergent [post-civilization] will be.

The reason for acceleration is:

• They can now build [patterning thinking-machines] which are strong at general pattern recognition and generation
• Most of the intellectual labour that humans do is just these two operations, specialized to particular domains
• (Indeed most human cognition is just these two operations, applied appropriately, e.g. to pattern recognition of valid reasoning steps)

It is therefore mostly the lack of major efforts, rather than technological inability, which prevents a majority of human intellectual work being automated.

(Some human intellectual work is automation-proof for now for economic reasons. Earth’s best [patterning-thinking-machines] are still significantly slower to learn and to generalize than humans are (in terms of the number of examples they need to see), so for specialized intellectual labour the training involved would currently be prohibitive.)

Over the next few economonths we can expect a few domains to start having large swathes of low-level work automated by [patterning thinking-machines]. This is most likely in areas that give human supervisors time to intervene in cases of errors (bemusingly, the humans have fixated on having thinking-machines take over the controls of dangerously-fast mobile machines, which have very little margin for error).

Humans working in these domains will increasingly devote time to training and overseeing these systems rather than doing the work themselves. (Eventually, oversight itself may be automated, but not in the first wave of applications.) In domains where this goes well, this will unlock much more labour than is currently provided by humans — usually until the point where it is no longer a significant bottleneck. The best humans will become wealthy training systems to excellence; many other humans will see demand for their services disappear or change significantly.

Typically humans will end up with access to much better teachers, expert advice, and (non-physical) assistants. Depending on how fast these various applications are developed, solar time might start to meaningfully slow^[3].
 

 

All of this will be of limited importance compared to the automation of research.

Research is a relatively hard thing to automate, but the problem will attract many top human minds. They will work to:

• Atomize the research process into steps that are within-grasp for well-trained [patterning thinking-machines]
• Record high-grade performance on these steps, to give the [patterning thinking-machines] something to emulate
• Work to supervise the output of the machines, both on their individual steps and the larger outputs of the research process^[4]

(Eventually, “end-to-end” systems aimed at good research will outperform these atomized versions. But such systems will be hungrier for training data, and take longer to train, so they won’t be the first high-performing research systems we see.)

Where will Earth civilization go from there? As research is significantly automated, solar time will slow, with a solar year taking more than an economonth and then likely several economonths. 

Overall, the direction of the world will be determined in significant part by the sequence in which important insights are arrived at.^[5] This in turn will be determined by which research areas are automated early, and well-resourced with expert supervision and [thinking-machines]. 

This is not simply a matter of economic incentives, although of course those matter. A relatively small number of decision-makers (the more far-sighted researchers/entrepreneurs/investors; very wealthy individuals or institutions; top researchers and entrepreneurs) could have significant influence on the paths unfolding.

 

The Omohundro-completeness threshold

It will around this time become technologically feasible to build [Omohundro-complete thinking-machines], which pursue [fully general power over the future] in service of their goals. Automating strategic thinking is of comparable technical difficulty to automating research (the major difficulties for both being holistic coherence, and evaluating highly novel ideas), but there will be less economic incentive to build [Omohundro-complete thinking-machines], so they will not appear as early as the automation of research. However, absent strong coordination to avoid building them (which humans do not presently seem capable of achieving), they will likely follow the automation of research by somewhere between 6 economonths and 6 econoyears. 

This matters. At the moment the strategic actors on Earth are all humans or political bodies composed of humans. The direction of civilization depends in real part on who is steering, and the advent of [Omohundro-complete thinking-machines] could greatly expand the set of possibilities. It is very possible that some [Omohundro-complete thinking-machines] created in the early days could — via their appointed successors — end up controlling much of that part of the universe which Earth-originating thinking beings can hope to claim.

The first powerful [Omohundro-complete thinking-machines] will most likely be cobbled together from more service-like systems, but could instead be constructed deliberately or emerge from some small constituent [reward-chasing-machine] seizing control of the larger system around it. Unfortunately it is too hard to predict at this stage: what systems might be created and how; what the priorities will be of those systems that acquire power; whether power transitions will be characterized by negotiation or expropriation.

 

Superhuman generalization

The next major foreseeable development is when the fundamental capacity of [thinking machines] to [learn/generalize] surpasses that of humans. Within an econoyear of this point, new [perspectives/paradigms] from direct observation will be more powerful than learning [perspectives/paradigms] from humans. Research automation using such [thinking-machines] will overtake research automation based on human paradigms. Solar time will slow further. Within the next few econoyears [thinking-machines] will advance until they vastly outperform human thinkers in all important intellectual tasks. If we take as a milestone the moment when humans represent less than a 1/144 part of at least 143/144 of important areas of intellectual work, this is likely to be passed between 2 and 12 econoyears after the start of significant research automation. (It is possible that it will occur before the emergence of [Omohundro-complete thinking-machines], although that would not necessarily be stabilizing.)

It is more likely than not that Earth will see the emergence of a coherently-coordinated [post-civilization] in the few econoyears immediately preceding or immediately following this milestone.

 

Anthropological note

These approaching developments have spurred relatively little interest among humans. Even those who do attend generally talk of “the time of transformative AI”, and mix up features of all of the above, as though there were one transition to come and not three.

It is hard to fully account for this, since humans are certainly capable of understanding three transitions, and have enough information to deduce their existence. It may be a combination of:

1. Few humans paying attention to this at all, because it is incongruent with their direct experience;
2. The expected slowdown of solar time, such that individual humans who observe one transition will typically observe subsequent transitions.^[6]

1. ^{^}

   A period of time over which the economy grows by a factor of e. One econoyear ago on Earth was 1989 in the local reckoning; two econoyears ago was 1962. Although theorists are dissatisfied with some unresolved issues in measuring the size of economies, pragmatists have won the battle over the most useful ways to discuss timelines of rising civilizations.

2. ^{^}

   Approximately the time over which the economy grows by one twelfth. There are twelve economonths in an econoyear (which pleasing fact informed my translation into English; although economonths and econoyears are currently much longer than solar months and years, this is unlikely to remain true for too long).

3. ^{^}

   i.e. relative to econotime.

4. ^{^}

   Since it is easier to recognise good research than to produce it; while in the end Goodhart’s Law will bite here as everywhere, there is a lot of value to be extracted before that becomes a serious problem.

5. ^{^}

   For example, insight into how to construct [[redacted]] before understanding how to safely structure access to this knowledge could end civilization abruptly. Or the relative degree of advancement of [Omohundro-complete thinking-machines] and insights into [direction-setting-for-thinking-machines] could determine the [goals/nature] of any subsequent emergent [post-civilization].

6. ^{^}

   The lifespan of an individual human has risen over their recorded history from around 1 economonth to around 3 econoyears; 12 econoyears is generally reckoned to be enough to see most civilizational transitions — at which point solar time will speed up again (until civilization approaches technological maturity).

12 comments

Comments sorted by top scores.