Douglas Hofstadter changes his mind on Deep Learning & AI risk (June 2023)?

Yeah, there's obviously SOME recursion there but it's still surprising that such a relatively low bandwidth recursion can still work so well. It's more akin to me writing down my thoughts and then rereading them to gather my ideas than the kind of loops I imagine our neurons might have.

That said, who knows, maybe the loops in our brain are superfluous, or only useful for learning feedback purposes, and so a neural network trained by an external system doesn't need them.

In what sense do they emulate these dynamics?

Further discussion on Twitter of feedforward vs recurrent.

Being an autoregressive language model is like having a strange form of amnesia, where you forget everything you thought about so far as soon as you utter a new word, and you can remember only what you said before.

that paper is one of many claiming some linear attention mechanism that's as good as full self attention. in practice they're all sufficiently much worse that nobody uses them except the original authors in the original paper, usually not even the original authors in subsequent papers.

the one exception is flash attention, which is basically just a very fancy fused kernel for the same computation (actually the same, up to numerical error, unlike all these "linear attention" papers).

>It turns out that using Transformers in the autoregressive mode (with output tokens being added back to the input by concatenating the previous input and the new output token, and sending the new versions of the input through the model again and again) results in them emulating dynamics of recurrent neural networks, and that clarifies things a lot...

I'll bite: Could you dumb down the implications of the paper a little bit, what is the difference between a Transformer emulating a RNN and  some pre-Transformer RNNs and/or not-RNN?

My much more novice-level answer to Hofstadter's intuition would have been: it's not the feedforward firing, but it is the gradient descent training of the model on massive scale (both in data and in computation). But apparently you think that something RNN-like about the model structure itself is important?

Yeah. I particularly hate the handwavium which makes this sound like it's super simple, just make the ASI, have it churn out labour for us, surely human society will adapt just nicely to the new state of things and chill. It's easy to say this if you think you're going to be the one in charge of the ASI because you're a CEO or big shot in some company (you may still be vastly overestimating your chances of controlling the ASI of course but that's just plain old hubris). But not so easy to believe it if instead you're more the kind of person who usually gets the short end of the stick. Like, as much as we may celebrate automation, it DOES often have hugely disruptive effects. There's a lot of pain hidden in that "some jobs are destroyed but others are created so it evens out". And that's not even a fraction of the pain that would be possible if ALL jobs were destroyed and never replaced and we would somehow have to find a way to deal with that.

And that's, again, just the most optimistic scenario. The more pessimistic one is more along the lines of "you're a megatherium and these strange new hairless apes with sticks have started flooding in from the north".

He was only a de facto mysterian: thought mind is so complicated that it may as well be mysterious (but ofc he believed it's ultimately just physics). This position is updateable, and he clearly updated.

Gwern's comment [LW(p) · GW(p)] makes it clear to me that Hofstadter has never been a mysterian. 

But only slightly. It appears that Hofstadter's doubts have been building for a long time in private, even to organizing informal conferences/meetings about it, to an extent that his op-eds don't convey (compare his comments in OP to his comments published in the Atlantic just a week before! they are so drastically different I was wondering if this was some sort of bizarre deepfake prank, but some cursory searching made it seemed legit and no one like Mitchell was saying it was fake and the text sounds like Hofstadter). On Twitter, John Teets helpfully notes that Mitchell has a 2019 book Artificial Intelligence: A Guide for Thinking Humans where she records some private Hofstadter material I was unfamiliar with:

Prologue: Terrified ...The meeting, in May 2014, had been organized by Blaise Agüera y Arcas, a young computer scientist who had recently left a top position at Microsoft to help lead Google’s machine intelligence effort...The meeting was happening so that a group of select Google AI researchers could hear from and converse with Douglas Hofstadter, a legend in AI and the author of a famous book cryptically titled Gödel, Escher, Bach: an Eternal Golden Braid, or more succinctly, GEB (pronounced “gee-ee-bee”). If you’re a computer scientist, or a computer enthusiast, it’s likely you’ve heard of it, or read it, or tried to read it...Chess and the First Seed of Doubt: The group in the hard-to-locate conference room consisted of about 20 Google engineers (plus Douglas Hofstadter and myself), all of whom were members of various Google AI teams. The meeting started with the usual going around the room and having people introduce themselves. Several noted that their own careers in AI had been spurred by reading GEB at a young age. They were all excited and curious to hear what the legendary Hofstadter would say about AI.

Then Hofstadter got up to speak. “I have some remarks about AI research in general, and here at Google in particular.” His voice became passionate. “I am terrified. Terrified.”

Hofstadter went on. [2. In the following sections, quotations from Douglas Hofstadter are from a follow-up interview I did with him after the Google meeting; the quotations accurately capture the content and tone of his remarks to the Google group.] He described how, when he first started working on AI in the 1970s, it was an exciting prospect but seemed so far from being realized that there was no “danger on the horizon, no sense of it actually happening.” Creating machines with human-like intelligence was a profound intellectual adventure, a long-term research project whose fruition, it had been said, lay at least “one hundred Nobel prizes away.” [Jack Schwartz, quoted in G.-C. Rota, Indiscrete Thoughts (Boston: Berkhäuser, 1997), pg22.] Hofstadter believed AI was possible in principle: “The ‘enemy’ were people like John Searle, Hubert Dreyfus, and other skeptics, who were saying it was impossible. They did not understand that a brain is a hunk of matter that obeys physical law and the computer can simulate anything … the level of neurons, neurotransmitters, et cetera. In theory, it can be done.” Indeed, Hofstadter’s ideas about simulating intelligence at various levels---from neurons to consciousness---were discussed at length in GEB and had been the focus of his own research for decades. But in practice, until recently, it seemed to Hofstadter that general “human-level” AI had no chance of occurring in his (or even his children’s) lifetime, so he didn’t worry much about it.

Near the end of GEB, Hofstadter had listed “10 Questions and Speculations” about artificial intelligence. Here’s one of them: “Will there be chess programs that can beat anyone?” Hofstadter’s speculation was “no.” “There may be programs which can beat anyone at chess, but they will not be exclusively chess players. They will be programs of general intelligence.”⁴

At the Google meeting in 2014, Hofstadter admitted that he had been “dead wrong.” The rapid improvement in chess programs in the 1980s and ’90s had sown the first seed of doubt in his appraisal of AI’s short-term prospects. Although the AI pioneer Herbert Simon] had predicted in 1957 that a chess program would be world champion “within 10 years”, by the mid-1970s, when Hofstadter was writing GEB, the best computer chess programs played only at the level of a good (but not great) amateur. Hofstadter had befriended Eliot Hearst, a chess champion and psychology professor who had written extensively on how human chess experts differ from computer chess programs. Experiments showed that expert human players rely on quick recognition of patterns on the chessboard to decide on a move rather than the extensive brute-force look-ahead search that all chess programs use. During a game, the best human players can perceive a configuration of pieces as a particular “kind of position” that requires a certain “kind of strategy.” That is, these players can quickly recognize particular configurations and strategies as instances of higher-level concepts. Hearst argued that without such a general ability to perceive patterns and recognize abstract concepts, chess programs would never reach the level of the best humans. Hofstadter was persuaded by Hearst’s arguments.

However, in the 1980s and ’90s, computer chess saw a big jump in improvement, mostly due to the steep increase in computer speed. The best programs still played in a very unhuman way: performing extensive look-ahead to decide on the next move. By the mid-1990s, IBM’s Deep Blue machine, with specialized hardware for playing chess, had reached the Grandmaster level, and in 1997 the program defeated the reigning world chess champion, Garry Kasparov, in a 6-game match. Chess mastery, once seen as a pinnacle of human intelligence, had succumbed to a brute-force approach.

Music: The Bastion of Humanity... Hofstadter had been wrong about chess, but he still stood by the other speculations in GEB...Hofstadter described this speculation as “one of the most important parts of GEB---I would have staked my life on it.”

I sat down at my piano and I played one of EMI’s mazurkas “in the style of Chopin.” It didn’t sound exactly like Chopin, but it sounded enough like Chopin, and like coherent music, that I just felt deeply troubled.

Hofstadter then recounted a lecture he gave at the prestigious Eastman School of Music, in Rochester, New York. After describing EMI, Hofstadter had asked the Eastman audience---including several music theory and composition faculty---to guess which of two pieces a pianist played for them was a (little-known) mazurka by Chopin and which had been composed by EMI. As one audience member described later, “The first mazurka had grace and charm, but not ‘true-Chopin’ degrees of invention and large-scale fluidity … The second was clearly the genuine Chopin, with a lyrical melody; large-scale, graceful chromatic modulations; and a natural, balanced form.” [ 6.  Quoted in D. R. Hofstadter, “Staring Emmy Straight in the Eye—and Doing My Best Not to Flinch,”† in Creativity, Cognition, and Knowledge, ed. T. Dartnell (Westport, Conn.: Praeger, 2002), 67–100.] Many of the faculty agreed and, to Hofstadter’s shock, voted EMI for the first piece and “real-Chopin” for the second piece. The correct answers were the reverse.

In the Google conference room, Hofstadter paused, peering into our faces. No one said a word. At last he went on. “I was terrified by EMI. Terrified. I hated it, and was extremely threatened by it. It was threatening to destroy what I most cherished about humanity. I think EMI was the most quintessential example of the fears that I have about artificial intelligence.”

Google and the Singularity: Hofstadter then spoke of his deep ambivalence about what Google itself was trying to accomplish in AI---self-driving cars, speech recognition, natural-language understanding, translation between languages, computer-generated art, music composition, and more. Hofstadter’s worries were underlined by Google’s embrace of Ray Kurzweil and his vision of the Singularity, in which AI, empowered by its ability to improve itself and learn on its own, will quickly reach, and then exceed, human-level intelligence. Google, it seemed, was doing everything it could to accelerate that vision. While Hofstadter strongly doubted the premise of the Singularity, he admitted that Kurzweil’s predictions still disturbed him. “I was terrified by the scenarios. Very skeptical, but at the same time, I thought, maybe their timescale is off, but maybe they’re right. We’ll be completely caught off guard. We’ll think nothing is happening and all of a sudden, before we know it, computers will be smarter than us.” If this actually happens, “we will be superseded. We will be relics. We will be left in the dust. Maybe this is going to happen, but I don’t want it to happen soon. I don’t want my children to be left in the dust.”

Hofstadter ended his talk with a direct reference to the very Google engineers in that room, all listening intently: “I find it very scary, very troubling, very sad, and I find it terrible, horrifying, bizarre, baffling, bewildering, that people are rushing ahead blindly and deliriously in creating these things.”

Why Is Hofstadter Terrified? I looked around the room. The audience appeared mystified, embarrassed even. To these Google AI researchers, none of this was the least bit terrifying. In fact, it was old news...Hofstadter’s terror was in response to something entirely different. It was not about AI becoming too smart, too invasive, too malicious, or even too useful. Instead, he was terrified that intelligence, creativity, emotions, and maybe even consciousness itself would be too easy to produce---that what he valued most in humanity would end up being nothing more than a “bag of tricks”, that a superficial set of brute-force algorithms could explain the human spirit.

As GEB made abundantly clear, Hofstadter firmly believes that the mind and all its characteristics emerge wholly from the physical substrate of the brain and the rest of the body, along with the body’s interaction with the physical world. There is nothing immaterial or incorporeal lurking there. The issue that worries him is really one of complexity. He fears that AI might show us that the human qualities we most value are disappointingly simple to mechanize. As Hofstadter explained to me after the meeting, here referring to Chopin, Bach, and other paragons of humanity, “If such minds of infinite subtlety and complexity and emotional depth could be trivialized by a small chip, it would destroy my sense of what humanity is about.”

...Several of the Google researchers predicted that general human-level AI would likely emerge within the next 30 years, in large part due to Google’s own advances on the brain-inspired method of “deep learning.”

I left the meeting scratching my head in confusion. I knew that Hofstadter had been troubled by some of Kurzweil’s Singularity writings, but I had never before appreciated the degree of his emotion and anxiety. I also had known that Google was pushing hard on AI research, but I was startled by the optimism several people there expressed about how soon AI would reach a general “human” level. My own view had been that AI had progressed a lot in some narrow areas but was still nowhere close to having the broad, general intelligence of humans, and it would not get there in a century, let alone 30 years. And I had thought that people who believed otherwise were vastly underestimating the complexity of human intelligence. I had read Kurzweil’s books and had found them largely ridiculous. However, listening to all the comments at the meeting, from people I respected and admired, forced me to critically examine my own views. While assuming that these AI researchers underestimated humans, had I in turn underestimated the power and promise of current-day AI?

...Other prominent thinkers were pushing back. Yes, they said, we should make sure that AI programs are safe and don’t risk harming humans, but any reports of near-term superhuman AI are greatly exaggerated. The entrepreneur and activist Mitchell Kapor advised, “Human intelligence is a marvelous, subtle, and poorly understood phenomenon. There is no danger of duplicating it anytime soon.” The roboticist (and former director of MIT’s AI Lab) Rodney Brooks agreed, stating that we “grossly overestimate the capabilities of machines---those of today and of the next few decades.” The psychologist and AI researcher Gary Marcus went so far as to assert that in the quest to create “strong AI”---that is, general human-level AI---“there has been almost no progress.”

I could go on and on with dueling quotations. In short, what I found is that the field of AI is in turmoil. Either a huge amount of progress has been made, or almost none at all. Either we are within spitting distance of “true” AI, or it is centuries away. AI will solve all our problems, put us all out of a job, destroy the human race, or cheapen our humanity. It’s either a noble quest or “summoning the demon.”

That is, whatever the snarky "don't worry, it can't happen" tone of his public writings about DL has been since ~2010, Hofstadter has been saying these things in private for at least a decade*, starting somewhere around Deep Blue which clearly falsified a major prediction of his, and his worries about the scaling paradigm intensifying ever since; what has happened is that only one of two paradigms can be true, and Hofstadter has finally flipped to the other paradigm (with ChatGPT-3.5 and then GPT-4 apparently being the straws that broke the camel's back). Mitchell, however, has heard all of this firsthand long before this podcast and appears to be completely immune to Hofstadter's concerns (publicly), so I wouldn't expect it to change her mind.

* I wonder what other experts & elites have different views on AI than their public statements would lead you to believe?

† Hofstadter's semi-mysterian/irreducible-complexity view:

So . . . chess-playing fell to computers? I don't feel particularly threatened or upset; after all, sheer computation had decades earlier fallen to computers as well. So a computer had outdone <a href="https://en.wikipedia.org/wiki/Daniel_Shanks">Daniel Shanks</a> in the calculation of digits of π—did it matter? Did that achievement in any way lower human dignity? Of course not! It simply taught us that calculation is more mechanical than we had realized. Likewise, <a href="https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)">Deep Blue</a> taught us that chess is more mechanical than we had realized. These lessons serve as interesting pieces of information about various domains of expertise, but to my mind they hardly seem to threaten the notion, which I then cherished and which I still cherish, that human intelligence is extraordinarily profound and mysterious.

It is not, I hasten to add, that I am a mystic who thinks that intelligence intrinsically resists implantation in physical entities. To the contrary, I look upon brains themselves as very complex machines, and, unlike <a href="https://en.wikipedia.org/wiki/John_Searle">John Searle</a> and <a href="https://en.wikipedia.org/wiki/Roger_Penrose">Roger Penrose</a>, I have always maintained that the precise nature of the physicochemical substrate of thinking and consciousness is irrelevant. I can imagine silicon-based thought as easily as I can imagine carbon-based thought; I can imagine ideas and meanings and emotions and a first-person awareness of the world (an "inner light", a "ghost in the machine") emerging from electronic circuitry as easily as from proteins and nucleic acids. I simply have always run on faith that when "genuine artificial intelligence" (sorry for the oxymoron) finally arises, it will do so precisely because the same degree of complexity and the same overall kind of abstract mental architecture will have come to exist in a new kind of hardware. What I do not expect, however, is that full human intelligence will emerge from something far simpler, architecturally speaking, than a human brain.

He explained a bunch of his position on this in Godel, Escher, Bach. If I remember correctly, it describes the limits of primitive recursive and general recursive functions this in chapter XIII. The basic idea (again, if I remember), is that a proof system can only reason about itself if its general recursive, and will always be able to reason about itself if its general recursive. Lots of what we see that makes humanity special compared to computers has to do with people having feelings and emotions and self-concepts, and reflection about past situations & thoughts. All things that really seem to require deep levels of recursion (this is a far shallower statement than what's actually written in the book). Its strange to us then that ChatGPT can mimic those same outputs with the only recursive element of its thought being that it can pass 16 bits to its next running.

I am working on human capability enhancement via genetics. I think it's quite plausible that we could create humans smarter than any that have ever lived within a decade. But even I think that digital intelligence wins in the end.

Like it just seems obvious to me. The only reason I'm even working in the field is because I think that enhanced humans could play an extremely critical role in the development of aligned AI. Of course this requires time for them to grow up and do research, which we are increasingly short of. But in case AGI takes longer than projected or we get our act together and implement a ban on AI capabilities improvements until alignment is solved, it still seems worth continuing the work to me.