Reflections on the state of the race to superintelligence, February 2025

post by Mitchell_Porter · 2025-02-23T13:58:07.663Z · LW · GW · 6 comments

Contents

6 comments

My model of the situation is that some time last year, the frontier paradigm moved from "scaling up large language models" to "scaling up chain-of-thought models". People are still inventing new architectures, e.g. Google's Titans, or Lecun's energy-based models. But it's conceivable that inference scaling really is the final paradigm prior to superintelligence. If it can produce von Neumann-level intelligence, then that should be the end, right? - the end, in the sense that control will pass out of human hands, unless there are a few humans embedded in the self-transformations of these genius-level AIs, and those embedded humans will become something more than human or other than human quickly enough. 

This leads to the concrete question, where are the new frontier models, the powerful chain-of-thought AIs, being produced? Because these organizations are also the contenders to produce the first superintelligence. My list of known or suspected organizations consists of 5 in the USA, 1 in China, and 1 in Israel. Of course there may be more. 

At the head of my list is Elon Musk's xAI. It's at the head of the list, not because I believe that Grok 3 is "the world's smartest AI", but because of its political advantages in Trump 2.0 America. xAI is part of the Musk family of companies, and these are now more deeply integrated into US government activities than ever before. There is a serious possibility that superintelligence will be "born" commanding all the resources of Musk's unprecedented empire, including social media, rocketry, robotics, brain-computer interfaces, and US government information systems. 

Intellectually, Trump 2.0 means the ascendancy to power of a large number of ideas from outside the institutional consensus of the liberal establishment (which includes at a minimum, the universities, the mainstream media, and all parts of the government): peace with Putin's Russia, RFK Jr's idea of health, downsizing the government, reversal of DEI policies, and probably more to come. In the relatively esoteric area of frontier AI policy, it is as if e/acc replaced effective altruism as the implicit zeitgeist. Concretely that seems to mean switching from regulation (of AI) and international treaties, to deregulation and a competitive race with China. 

Again xAI has a bit of an advantage here, because Musk made X-Twitter friendly to these outsider ideas, in advance of Trump 2.0's right-wing populist revolution seizing institutional power. On the other hand, the other American AI companies were comfortable with Biden-era liberal progressivism, and have had to reorient themselves to the new order. I'll get to that in a moment, but first I'll address the situation of the second company in my list, OpenAI, which has an extra problem in addition to the change in political paradigm. 

Sam Altman is doing what he can to keep up - he wasn't at the inauguration, but literally the next day he was co-hosting the launch of the Stargate project with Trump. Altman's real problem is his beef with Elon Musk, who evidently wants to stop or suborn OpenAI, and has a lot of resources with which to do that. A year ago I would have said that OpenAI can draw on the resources of Microsoft to defend itself (since the alliance with Microsoft is what allowed Altman to hang on as CEO in November 2023), but I'm not up-to-date on OpenAI's partnerships, e.g. I think I heard of some partnership with Amazon too. 

Politics aside, I think OpenAI may be the technical leader, with GPT-5 perhaps coming later this year and incorporating the best of o1 and o3. 

Third and fourth on my list are Anthropic (with Claude) and Google (with Gemini). Politically I see them as pragmatically going with the flow - I think both Dario Amodei and Demis Hassabis have given lip service to the idea that democracy must get to superintelligence before authoritarianism does, while also reminding us that superintelligence in itself is dangerous if unaligned. 

Technically, Anthropic might be the leader in AI safety, since they took on Jan Leike and the rest of OpenAI's superalignment team, while Google is a ubiquitous behemoth with vast resources, the IBM of the Internet era, and has both the advantages and disadvantages that come with this. 

Fifth on my list is a hypothetical organization: The Project, Leopold Aschenbrenner's name for an American Manhattan Project aiming to create superintelligence. We don't know that it exists; we only have speculation about AI researchers who quit in order to pursue unknown exciting opportunities. If it does exist, it is also possible that it exists under the umbrella of one or more of the companies already in the list. 

That's my list for America. Now, onto my final two contenders. These are DeepSeek in China, and Ilya Sutskever's opaque Safe Superintelligence Inc., which is divided between Palo Alto and Tel Aviv. I won't speculate about their political context except to note that they exist outside or half-outside the American scene. 

Given this situation, in which there are at least six separate centers of research that are in with a chance of creating superintelligence, and dozens, possibly hundreds more worldwide, that are either doing it or want to do it; and given the fact that I am not part of any of those research centers; my strategy for trying to increase the chance of a human-friendly outcome, is to contribute to the public discussion of how to make autonomous superintelligence "ethical" or "human-friendly" or "superaligned", since the public discussion can in principle be noticed by any of the participants in the AI race, and if there are good valid ideas, they just might take note and implement them. (For example, at the moment I'm interested in what happens if you combine Joshua Clymer's new thoughts on safely outsourcing AI alignment tasks to AI [LW · GW], with June Ku's old CEV-like proposal at MetaEthical.AI - does it take us far beyond Eliezer's own thoughts on "Interim Friendliness" from the early 2000s?) 

It's true that there's an unknown number of unsolved questions remaining to be answered, in the theory and practice of safe superintelligence. The situation we've arrived at, in which the risks inherent to the creation of superintelligence are barely publicly acknowledged by the protagonists of the race, is far from desirable. It would be best to cross that threshold only once you really know what you're doing; but that is not the attitude that has prevailed. 

However, I don't consider it impossible that the theory will actually be figured out. Knowledge is distributed highly unequally in the world. Individuals and groups with extreme expertise do exist; and humanity plausibly has the foundations needed to figure out the theory of safe superintelligence. Our best experts do have an advanced understanding of quite a lot of physics, mathematics, and computation; and the people making the frontier models do at least know what recipes they are using to create and manage their AIs. Topics like consciousness and (meta)philosophy are a bit more problematic, but we do have a lot of ideas and data to work with. 

And finally, the source of our peril - the rapid climb in AI capabilities towards superintelligence - also means that all kinds of hard problems may be solved at unprecedented speed, even before we cross the threshold. So I choose to stay engaged, add my thoughts to the greater flow, and hope that somewhere, the problems that need to be solved will actually be solved. 

6 comments

Comments sorted by top scores.