What would it look like if it looked like AGI was very near?

Note:  everything stated about 2021 and earlier is actually the case in the real world; everything stated about the post-2021 world is what I'd expect to see contingent on this scenario being true, and something I would give decently high probabilities of in general.  I believe there is a fairly high chance of AGI in the next 10 years. 

12 July 2031, Retrospective from a post-AGI world:

By 2021, it was blatantly obvious that AGI was immanent.  The elements of general intelligence were already known:  access to information about the world, the process of predicting part of the data from the rest and then updating one's model to bring it closer to the truth (note that this is precisely the scientific method, though the fact that it operates in AGI by human-illegible backpropagation rather than legible hypothesis generation and discarding seems to have obscured this fact from many researchers at the time), and the fact that predictive models can be converted into generative models by reversing them:  running a prediction model forwards predicts levels of X in a given scenario, but running it backwards predicts which scenarios have a given level of X.  A sufficiently powerful system with relevant data, updating to improve prediction accuracy and the ability to be reversed to generate optimization of any parameter in the system is a system that can learn and operate strategically in any domain.  

Data wasn't exactly scarce in 2021.  The internet was packed with it, most of it publicly available, and while the use of internal world-simulations to bootstrap an AI's understanding of reality didn't become common in would-be general programs until 2023, it was already used in more narrow neural nets like AlphaGo by 2016; certainly researchers at the time were already familiar with the concept.  

Prediction improvement by backpropagation was also well known by this point, as was the fact that this is the backbone of human intelligence.  While there was a brief time when it seemed like this might be fundamentally different than the operation of the brain (and thus less likely to scale to general intelligence) given that human neurons only feed forwards, it was already known by 2021 that the predictive processing algorithm used by the human neocortex is mathematically isomorphic to backpropagation, albeit implemented slightly differently due to the inability of neurons to feed backwards.  

The interchangeability of prediction and optimization or generation was known as well, indeed it wasn't too uncommon to use predictive neural nets to produce images (one not-uncommon joke application was using porn filters to produce the most pornographic images possible according to the net), and the rise of DeepMind's complementary AIs DALL-E (image from text) and CLIP (text from image) showed the interchangeability in a striking way (though careful observers might note that CLIP wasn't reversed DALL-E; the twin nets merely demonstrated that the calculation can go either way; the reversed porn filter was a more rigorous demonstration of optimization from prediction).

Given that all the pieces for AGI thus existed in 2021, why didn't more people realize what was coming?  For that matter, given that all the pieces existed already, why did true AGI take until 2023, and AGI with a real impact on the world until 2025?  The answer to the second question is scale.  All animal brains operate on virtually identical principles (though there are architectural differences, e.g. striatum vs pallium), yet the difference between a human and a chimp, let alone a human and a mouse, is massive.  Until the rise of neural nets, it was commonly assumed that AGI would be a matter primarily of more clever software, rather than simply scaling up relatively simple algorithms.  The fact that greater performance is primarily the result of simple size, rather than brilliance on the part of the programmers even became known as the Bitter Lesson, as it wasn't exactly easy on designers' egos.  With the background assumption of progress as a function of algorithms rather than scale, it was easy to miss that AlphaGo already had nearly everything a modern superintelligence needs; it was just small.  

From 2018 through 2021, neural nets were built at drastically increasing scales.  GPT (2018) had 117 million parameters, GPT-2 (2019) had 1.5 billion, GPT-3 (2020) had 175 billion, ZeRO-Infinity (2021) had 32 trillion.  By comparison to animal brains (a neural net's parameter is closely analogous to a brain's synapse), that is similar to an ant (very wide error bars on this one; on the other comparisons I was able to find synapse numbers, but for an ant I could only find the number of neurons), bee, mouse and cat respectively.  Extrapolating this trend, it should not have been hard to see human-scale nets coming (100 trillion parameters, reached by 2022), nor AIs orders of magnitude more powerful than this.

Moreover, neural nets are much more powerful in many ways than their biological counterparts.  Part of this is speed (computers can operate around a million times faster than biological neurons), but a more counterintuitive part of this is encephalization.  Specifically, the requirements of operating an animal's body are sufficiently intense that available intelligence for other things scales not with brain size, but with the ratio of brain size to body size, called the encephalization quotient (this is why elephants are not smarter than humans, despite having substantially larger brains).  An artificial neural net, of course, is not trying to control a body, and can use all of its power on the question at hand.  This allowed even a relatively small net like GPT-3 to do college-level work in law and history by 2021 (subjects that require actual understanding remained out of reach of neural nets until 2022, though the 2021 net Hua Zhibing, based on the Chinese Wu Dao 2.0 system, came very close).  Given that a mouse-sized neural net can compete with college students, it should have been clear that human-sized nets would posses most elements of general intelligence, and the nets that soon followed at ten and one hundred times the scale of the human brain would be capable of it.  

Given the (admittedly retrospective) obviousness of all this, why wasn't it widely recognized at the time?  As previously stated, much of the lag in recognition was driven by the belief that progress would be driven more by advances in algorithms than by scale.  Given this belief, AGI would appear extraordinarily difficult, as one would try to imagine algorithms capable of general intelligence at small scales (DeepMind's AlphaOmega proved in 2030 that this is mathematically impossible; you can't have true general intelligence at much below cat-scale, and it's very difficult to have it below human-scale!)  Even among those who understood the power of scaling, the fact that it's almost impossible to have AI do anything in the real world beyond very narrow applications like self-driving cars without reaching the general intelligence threshold made it appear plausible that simply building larger GPT-style systems wouldn't be enough without another breakthrough.  However, in 2021 DeepMind published a landmark paper entitled "Reward is Enough", recognizing that reward-based reinforcement learning was in fact capable of scaling to general intelligence.  This paper was the closest thing humanity ever got to a fire alarm for general AI:  a fairly rigorous warning that existing models could scale up without limit, and that AGI was now only a matter of time, rather than requiring any further real breakthroughs.  

After that paper, 2022 brought human-scale neural nets (not quite fully generally intelligent, due to lacking human instincts and only being trained on internet data, which leaves some gaps that require substantially superhuman capacity to bridge through inference alone), and 2023 brought the first real AGI, with a quadrillion parameters, powerful enough to develop an accurate map of the world purely through a mix of internet data and internal modeling to bootstrap the quality of its predictions.  After that, AI was considered to have stalled, as alignment concerns prohibited the use of such nets to optimize the real world, until 2025 when a program that trained agents on modeling each others' full terminal values from limited amounts of data allowed the safe real-world deployment of large-scale neural nets.  Mankind is eternally grateful to those who raised the alarm about the value alignment problem, without which DeepMind would not have conducted that crucial hiatus, and without which our entire light cone would now be paperclips (instead of just the Horsehead Nebula, which Elon Musk converted to paperclips as a joke).

This is what I would expect an AGI takeoff to look like if we are in fact in a "hardware overshoot".  I actually think a hardware-bound "slow takeoff" is more likely, but I'd put a scenario like this at >5%.

I should have known that AGI was near the moment that BetaStar was released.  Unlike AlphaStar, which was trained using more compute than any previous algorithm and still achieved sub human-expert performance, BetaStar was trained by a researcher on a single TPU  in under a month and could beat the world's best player even when limited to 1/2 of the actions-per-minute of human players.  Unlike AlphaStar, which used a swarm of Reinforcement Learners to learn a strategy, BetaStar used a much more elegant algorithm that could be said to be  a combination of Transformers (of GPT-3 fame) and good-old-fashioned AB-pruning (the same algorithm used by DeepBlue 30 years ago).  

The trick was finding a way to combine these that didn't result in a combinatorial explosion.  Not only did the trick work, but because transformers were known to work on a wide variety of domains (text, images, audio, video, gestures,...), it was immediately obvious how to apply the BetaStar  algorithm to literally every domain.  Motion-planning for robots, resume writing, beating the stock market.

Even if I didn't see it coming, the experts a Google, OpenAI, and all of the  world's major governments did.  Immediately a world-wide arms race was  launched to see who could scale BetaStar up as fast as possible.  First place meant ruling the world.  Second place meant the barest chance at survival.  Third place meant extinction.

OpenAI was the first to announce that they had trained a version of BetaStar that appeared to have the intelligence of a 5-year-old child.  A week later Google announced that their version of BetaStar was equivalent of a PhD Grad Student.  The NSA didn't say how smart their version of BetaStar was.  Rather, the president of the United States announced that every single super-computer and nuclear-weapon in China, Russia, Iran, North Korea and Syria had been destroyed.

A few weeks later, ever single American received a check for $10,000 and a letter explaining that the checks would keep coming every month thereafter.  A few riots broke out around the world in resistance to "American Imperialism", but after checks started arriving in other countries, most people stopped complaining.

Nobody really knows what the AI is up to these days, but life on Earth is good so far and we try not to worry about it.  Space, however, --much to Elon Musk's disappointment-- belongs to the AI.

Signs we are in hardware overshoot:

1. A novel algorithm achieves state-of-the-art performance on a well-studied problem using 2-3 orders of magnitude less compute
2. It is apparent to experts how this algorithm generalizes to other real-world problems.
3. Major  institutions undertake a "Manhattan Project" style arms-race to scale up a general-purpose algorithm.

Caveats

I gave this story a "happy ending".  Hardware overshoot (and other forms of fast AGI takeoff) is the most-dangerous version of AGI because it has the ability to quickly surpass all human beings.  It's easy to imagine a version of the story where the winner of the arms race is not benevolent, or where there is an alignment-failure and humans lose control of the AGI entirely.

It was a slippery slope, with those Neural Networks. They were able to do more and more things, previously unimagined to be possible for them. It was a big surprise for everyone, how good they were at chess, 3600 or so Elo points. Leela Chess Zero invented some theoretical breakthroughs, soon to be exploited by more algorithmic, non-NN chess engines like Stockfish, for its position evaluation function. Even back then, I was baffled by people expecting that this propagation will soon stop, due to some unexpected effect, which never came. Not in chess, nor anywhere else.

It was indeed a matter of "when", not of "maybe not" anymore. Yes, those first mighty AI's were quite fake, they have no real clue. Except that this mattered less and less and it was less and less true. In an increasing number of fields. 

It was only a matter of time when the first translators from the gibberish weight tables learned by NN's, to exact algorithms will emerge. Something which people have previously done, stealing ideas from Leela, implementing them with rigor into algorithmic schemes of Stockfish -- AI learned as well. Only better, of course.

By then, the slope was very slippery, indeed. I still can't comprehend, how this wasn't clear to everyone, even back then, less than 10 years ago.

I never got around to the scenario-based planning post, but things sure have changed in 10 months!

I wrote about this from a retrospective perspective already. "If computer power is the only thing standing between us and the singularity then we will finally have enough computer power... a decade ago." Humans have a slight advantage in compute architecture now, but I doubt that's enough to overcome computers' other advantages.

https://www.lesswrong.com/posts/m5rvZBKyMRtFo53wZ/hardware-is-already-ready-for-the-singularity-algorithm