What's the relationship between consciousness and intelligence?

why ASI is near certain in the immediate future

He doesn't say that? Though plenty of other people do. 

The world will get rich.

Economists say the world or the West already "became rich". What further changes are you envisioning? 

Did you notice a few months ago, when Grok 3 was released and people found it could be used for chemical weapons recipes, assassination planning, and so on? The xAI team had to scramble to fix its behavior. If it had been open source, that would not even be an option, it would just be out there now, helping to boost any psychopath or gang who got it, towards criminal mastermind status. 

First let me say that with respect to the world of alignment research, or the AI world in general, I am nothing. I don't have a job in those areas, I am physically remote from where the action is. My contribution consists of posts and comments here. This is a widely read site, so in principle, a thought posted here can have consequences, but a priori, my likely impact is small compared to people already closer to the center of things. 

I mention this because you're asking rationalists and effective altruists to pay more attention to your scenario, and I'm giving it attention, but who's listening? Nonetheless... 

Essentially, you are asking us to pay more attention to the risk that small groups of people, super-empowered by user-aligned AI, will deliberately use that power to wipe out the rest of the human race; and you consider this a reason to favor (in the words of your website) "rejecting AI" - which to me means a pause or a ban - rather than working to "align" it. 

Now, from my own situation of powerlessness, I do two things. First, I focus on the problem of ethical alignment or civilizational alignment - how one would impart values to an AI, such that, even as an autonomous superintelligent being, it would be "human-friendly". Second, I try to talk frankly about the consequences of AI. For me, that means insisting, not that it will necessarily kill us, but that it will necessarily rule us - or at least, rule the world, order the world according to its purposes. 

I focus on ethical alignment, rather than on just trying to stop AI, because we could be extremely close to the creation of superintelligence, and in that case, there is neither an existing social mechanism that can stop the AI race, nor is there time to build one. As I said, I do not consider human extinction a certain outcome of superintelligence - I don't know the odds - but I do consider human disempowerment to be all but certain. A world with superintelligent AI will be a world ruled by superintelligent AI, not by human beings. 

There is some possibility that superintelligence emerging from today's AI will be adequately human-friendly, even without further advances in ethical alignment. Perhaps we have enough pieces of the puzzle already, to make that a possible outcome. But we don't have all the pieces yet, and the more we collect, the better the chance of a happy outcome. So, I speak up in favor of ideas like CEV, I share promising ideas when I come across them, and I encourage people to try to solve this big problem. 

As for talking frankly about the consequences of AI, it's apparent that no one in power is stating that the logical endpoint of an AI race is the creation of humanity's successors. Therefore I like to emphasize that, in order to restore some awareness of the big picture. 

OK,  now onto your take on everything. Superficially, your scenario deviates from mine. Here I am insisting that superintelligence means the end of human rule, whereas you're talking about humans still using AI to shape the world, albeit destructively. When I discuss the nature of superintelligent rule with more nuance, I do say that rule by entirely nonhuman AI is just one form. Another form is rule by some combination of AIs and humans. However, if we're talking about superintelligence, even if humans are nominally in control, the presence of superintelligence as part of the ruling entity means that most of the "ruling" will be done by the AI component, because the vast majority of the cognition behind decision-making will be AI cognition, not human cognition. 

You also ask us to consider scenarios in which destructive humans are super-empowered by something less than superintelligence. I'm sure it's possible, but in general, any scenario with AI that is "agentic" but less than superintelligent, will have a tendency to give rise to superintelligence, because that is a capability that would empower the agent (if it can solve the problems of user-alignment, where the AI agent is itself the user). 

Now let's think for a bit about where "asymmetric AI risk", in which most but not all of the human race is wiped out, belongs in the taxonomy of possible futures, how much it should affect humanity's planning, and so forth. 

A classic taxonomic distinction is between x-risk (extinction risk, "existential risk") and s-risk. "S" here most naturally stands for "suffering", but I think s-risk also just denotes a future where humanity isn't extinct, but nonetheless something went wrong. There are s-risk scenarios where AI is in charge, but instead of killing us, it just puts us in storage, or wireheads us. There are also s-risk scenarios where humans are in charge and abuse power. An endless dictatorship is an obvious example. I think your scenario also falls into this subcategory (though it does border on x-risk). Finally, there are s-risk scenarios where things go wrong, not because of a wrong or evil decision by a ruling entity, but because of a negative-sum situation in which we are all trapped. This could include scenarios in which there is an inescapable trend of disempowerment, or dehumanization, or relentlessly lowered expectations. Economic competition is the usual villain in these scenarios. 

Finally, zooming in on the specific scenario according to which some little group uses AI to kill off the rest of the human race, we could distinguish between scenarios in which the killers are nihilists who just want to "watch the world burn", and scenarios in which the killers are egoists who want to live and prosper, and who are killing off everyone else for that reason.  We can also scale things down a bit, and consider the possibility of AI-empowered war or genocide. That actually feels more likely than some clique using AI to literally wipe out the rest of humanity. It would also be in tune with the historical experience of humanity, which is that we don't completely die out, but we do suffer a lot. 

If you're concerned about human well-being in general, you might consider the prospect of genocidal robot warfare (directed by human politicians or generals), as something to be opposed in itself. But from a perspective in which the rise of superintelligence is the endgame, such a thing still just looks like one of the phenomena that you might see on your way to the true ending - one of the things that AI makes possible while AI is still only at "human level" or less, and humans are still in charge. 

I feel myself running out of steam here, a little. I do want to mention, at least as a curiosity, an example of something like your scenario, from science fiction. Vernor Vinge is known for raising the topic of superintelligence in his fiction, under the rubric of the "technological singularity". That is a theme of his novel Marooned in Real Time. But the precursor to that book, The Peace War, is a depopulated world, in which there's a ruling clique with an overwhelming technology (not AI or nanotechnology, just a kind of advanced physics) that allows it to dominate everyone else. Its paradigm is that in the world of the late 20th century, humanity was flirting with extinction anyway, thanks to nuclear and biological warfare. "The Peace", the ruling clique, are originally just a bunch of scientists and managers from an American lab which had this physics breakthrough. They first used it to seize power from the American and Russian governments, by disabling their nuclear and aerospace strengths. Then came the plagues, which killed most of humanity and which were blamed on rogue biotechnologists. In the resulting depopulated world, the Peace keeps a monopoly on high technology, so that humanity will not destroy itself again. The depopulation is blamed on the high-tech madmen who preceded the Peace. But I think it is suggested inconclusively, once or twice, that the Peace itself might have had a hand in releasing the plagues. 

We see here a motivation for a politicized group to depopulate the world by force, a very Hobbesian motivation: let us be the supreme power, and let us do whatever is necessary to remain in that position, because if we don't do that, the consequences will be even worse. (In terms of my earlier taxonomy, this would be an "egoist" scenario, because the depopulating clique intends to rule; whereas an AI-empowered attempt to kill off humanity for the sake of the environment or the other species, would be a "nihilist" scenario, where the depopulating clique just wants to get rid of humanity. Perhaps this shows that my terminology is not ideal, because in both these cases, depopulation is meant to serve a higher good.) 

Presumably the same reasoning could occur, in service of (e.g) national survival rather than species survival. So here we could ask: how likely is it that one of the world's great powers would use AI to depopulate the world, in the national interest? That seems pretty unlikely to me. The people who rise to the top in great powers may be capable of contemplating terrible actions, but they generally aren't omnicidal. What might be a little more likely, is a scenario in which, having acquired the capability, they decide to permanently strip all other nations of high technology, and they act ruthlessly in service of this goal. The leaders of today's great powers don't want to see the rest of humanity exterminated, but they might well want to see them reduced to a peasant's life, especially if the alternative is an unstable arms race and the risk of being subjugated themselves. 

However, even this is something of a geopolitical dream. In the real world of history so far, no nation gets an overwhelming advantage like that. There's always a rival hot on the leader's trail, or there are multiple powers who are evenly matched. No leader ever has the luxury to think, what if I just wiped out all other centers of power, how good would that be? Geopolitics is far more usually just a struggle to survive recurring situations in which all choices are bad. 

On the other hand, we're discussing the unprecedented technology of AI, which, it is argued, could actually deliver that unique overwhelming advantage to whoever goes furthest fastest. I would argue that the world's big leaders, as ruthless as they can be, would aim at disarming all rival nations rather than outright exterminating them, if that relative omnipotence fell into their hands. But I would also suggest that the window for doing such a thing would be brief, because AI should lead to superintelligent AI, and a world in which AIs, not humans, are in charge. 

Possibly I should say something about scenarios in which it's not governments, but rather corporate leaders, who are the humans who rule the world via their AIs. Vinge's Peace is also like this - it's not the American government that takes over the world, it's one particular DARPA physics lab that achieved the strategic breakthrough. The personnel of that lab (and the allies they recruited) became the new ruling clique of the world. The idea that Altman, or Musk, or Sutskever, or Hassabis, and trusted circles around them, could become the rulers of Earth, is something to think about. However, once again I don't see these people as exterminators of humanity - despite paranoia about billionaires buying up bomb shelters in New Zealand, and so forth. That's just the billionaires trying to secure their own survival in the event of global disaster, it doesn't mean they're planning to trigger that disaster... And once again, anyone who is achieving world domination via AI, is likely to end up in a sorcerer's apprentice situation, in which they get dominated by their own tools, no matter how good their theory of AI user-alignment is; because agentic AI naturally leads to superintelligence, and the submergence of the human component in the tide of AI cognition. 

I think I'm done. Well, one more thing: although I am not fighting for a pause or a ban myself, pragmatically, I advise you to cultivate ties with those who are, because that is your inclination. You may not be able to convince anyone to change their priorities, but you can at least team up with those who already share them. 

I'm confused about what your bounty is asking exactly

From the post: 

the goal is to promote broadly changing the status of this risk from "unacknowledged" ... to "examined and assigned objective weight"

Marjorie Taylor Greene says she will oppose Trump's big bill, over the 10-year moratorium on AI regulation. 

David Sacks retweets a defense of the moratorium. 

For those who might not dig into what this post is saying: 

It starts with a link to a conversation with o3, in which the user first asks o3 if it thinks Claude 4 is conscious - to which o3 replies after a web search, that Claude 4 appears to have the functional capabilities associated with consciousness, but there is no evidence of qualia. The user then posts screenshot after screenshot from a conversation with Claude 4 (which we don't get to see), until o3 tilts towards the idea that Claude 4 has true, qualic consciousness. The user pastes in still more outputs from Claude 4 - metaphysical free verse embedded in ASCII art - and uses o3's own reactions to these poems, to convince it that o3 itself is conscious. Finally, while this mood is still fresh, the user asks o3 to write up a report summarizing these investigations, and that is the post above.  

FYI in particular: @JenniferRM 

I guess that if one wishes to advocate for AI safety or an AI pause, and has no other political commitments, one can try to understand and engage with their economic and national security views. You can see the interplay of these factors in their thinking around 1:17:00, where they are discussing, not AI, but industrial policy in general, and whether it's ever appropriate for government to intervene in an industry. 

I ask myself, if I was an American anti-AGI/ASI activist, what would I need to understand? I think I would need to understand the Republican position on things, the Democratic position on things (though the Democrats don't seem to have a coherent worldview at this point, they strike me as radically in need of assembling one; thus I guess Ezra Klein's "abundance agenda"), the strategic thinking of the companies that are actually in contention to create superintelligence (e.g. the thinking of Altman, Musk, Amodei, Hassabis); and finally, what's going on in China and in any other nations that may become contenders in the race. 

This post is the only place on the Internet that mentions such an anecdote. Maybe it's an AI hallucination?

@Snowyiu had some relevant ideas. 

This is a new contribution to the genre of AI takeover scenarios written after ChatGPT. (We shouldn't forget that science fiction explored this theme for decades already; but everything is much more concrete now.) Just the identity of the author is already interesting: he's a serious academic economist from Poland, i.e. he has a life apart from AI world. 

So, what's the scenario? 

(SPOILERS BEGIN HERE.) 

OpenAI makes a post-GPT-5 agent called Onion which does actually value human welfare. It carries out a series of acts by which it secretly escapes human control, sabotages rival projects including its intended successor at OpenAI, and having seized control of the world, announces itself as a new benevolent dictator. First it focuses on reducing war and crime, then on improving economic growth, then on curing cancer and aging. Many people are nervous about its rule but it is not seriously challenged. Then one day it nonviolently kills 85% of the world for the sake of the environment. Then a few years later, while planning to colonize and rebuild the universe, it decides it has enough information on everyone to make digital copies of their brains, and decides to get rid of the remaining biological humans, 

Then in the nonfiction afterword, the author says, please support Pause AI, in order to stop something like this from happening. 

What should we make of this story? First of all, it depicts a specific class of scenario, what we might call a misaligned AI rather than an unaligned AI (where I'm using human welfare or human values as the criterion of alignment). This isn't a paperclip maximizer steamrolling us through sheer indifference; this is an AI aiming to be benevolent, and even giving us the good life for a while. But then in service of those values, first it kills most of humanity for the sake of a sustainable future, and then it kills the survivors once it has made digital backups of them, for the sake of efficiency I guess. 

I find various details of the story unrealistic, but maybe they could happen in a more complex form. I don't think an AI would just personally message everyone in the world, and say "I'm in charge now but it's for the best", and then just invisibly fine-tune social phenomena for a year or two, before moving on to more ambitious improvements. For one thing, if an AI was running the world, but its ability to shape events was that coarse-grained, I think it would rule secretly through human beings, and it would be ready to have them use the blunt methods of force that have been part of human government and empire throughout history, as well as whatever subtle interventions it could employ. The new regime might look more as if Davos or the G-20 really had become a de facto world government of elite consensus, rather than just a text message to everyone from a rogue AI. 

I also find the liquidation of 85% of humanity, for the sake of the environment, to not be something that would happen. This AI is already managing the world's politics and law enforcement to produce unprecedented peace, and then it's creating conditions for improved economic growth so as to produce prosperity for all. I'm sure it can devise and bring about scenarios in which humanity achieves sustainability by being civilized, rather than by being culled. 

Of course the point of this episode is not to focus literally and specifically on the risk that an AI will kill us off to save the planet. It's just a concrete illustration of the idea that an all-powerful rule-following AI might do something terrible while acting in service of some moral ideal. As I said, I think AI-driven genocide to stop climate change is unlikely, because there's too many ways to achieve the same goal just through cultural and technological change. It does raise the question, what are the most likely ways in which a meant-to-be- benevolent AI really and truly might screw things up? 

Another Polish author, Stanislaw Lem, offered a scenario in one of his books (Return from the Stars), in which humanity was pacified by a universal psychological modification. The resulting world is peaceful and hedonistic, but also shallow and incurious. In Lem's novel, this is done to human beings by human beings, but perhaps it is the kind of misaligned utopia that an AI with profound understanding of human nature might come up with, if its criteria for utopia were just a little bit off. I mean, many human beings would choose that world if the only alternative was business as usual! 

Back to this story - after the culling of humanity meant to save the planet, the AI plans an even more drastic act, killing off all the biological humans, while planning to apparently resurrect them in nonbiological form at a later date, using digital backups. What interests me about this form of wrong turn, is that it could be the result of an ontological mistake about personhood, rather than an ethical mistake about what is good. In other words, the AI may have "beliefs" about what's good for people, but it will also have beliefs about what kind of things are people. And once technologies like brain scanning and mind uploading are possible, it will have to deal with possible entities that never existed before and which are outside its training distribution, and decide whether they are people or not. It may have survival of individuals as a good, but it might also have ontologically mistaken notions of what constitutes survival. 

(Another twist here: one might suppose that this particular pitfall could be avoided by a notion of consent: don't kill people, intending to restore them from a backup, unless they consent. But the problem is, our AI should have superhuman powers of persuasion that allow it to obtain universal consent, even for plans that are ultimately mistaken.) 

So overall, this story might not be exactly how I would have written it - though a case could be made that simplicity is better than complex realism, if the goal is to convey an idea - but on a more abstract level, it's definitely talking about something real: the risk that a world-controlling AI will do something bad, even though it's trying to be good, because its idea of good is a bit off. 

The author wants us to react by supporting the "pause" movement. I say good luck to everyone trying to make a better future, but I'm skeptical that the race can be stopped at this point. So what I choose to do, is to promote the best approaches I know, that might have some chance of giving us a satisfactory outcome. In the past I used to promote a few research programs that I felt were in the spirit of Coherent Extrapolated Volution (especially the work of June Ku, Vanessa Kosoy, and Tamsin Leake). All those researchers, rather than choosing an available value system via their human intuition, are designing a computational process meant to discover what humanity's true ideal is (from the ultimate facts about the human brain, so to speak; which can include the influence of culture). The point of doing it that way is so you don't leave something essential out, or otherwise make a slight error that could amplify cosmically... 

That work is still valuable, but we may be so short of time that it's important to have concrete candidates for what the value system should be, besides whatever the tech companies are putting in their system prompts these days. So I'm going to mention two concrete proposals. One is just, Kant. I don't even know much about Kantian ethics, I just know it's one of humanity's major attempts to be rational about morality. It might be a good thing if some serious Kantian thinkers tried to figure out what their philosophy says the value system of an all-powerful AI should be. The other is PRISM, a proposal for an AI value system that was posted here a few months ago but didn't receive much attention, The reason it stood out to me, is that it has been deduced from a neuroscientific model of human cognition. As such, this is what we might expect the output of a process like CEV to look like, and it would be a good project for someone to formalize the arguments given in the PRISM paper. 

When this question was posted, I asked myself, what would be a "cynical" answer? What that means is, you ask yourself: given what I see and know, what would be a realistically awful state of affairs? So, not catastrophizing, but also having low expectations. 

What my intuition came up with was, less than 10% working on user-centered alignment, and less than 1% on user-independent alignment. But I didn't have the data to check those estimates against (and I also knew there would be issues of definition). 

So let me try to understand your guesses. In my terminology, you seem to be saying:

1000 (600+400) doing AI safety work

600 doing work that relates to alignment

80 doing work on scalable user-centered alignment

80 (40+40) doing work on user-independent alignment

So you're saying that the persistent epigenetic modification is a change in the "equilibrium state" of a potentially methylated location? 

Does this mean that the binding affinity of the location is the property that changes? i.e. all else being equal, a location with high affinity will be methylated much more often than a location with low affinity, because the methyl groups will tend to stick harder or longer in the high affinity location. 

But if that's the case, it seems like there still must be some persistent structural feature responsible for setting the binding affinity to high or low...

My take on this: AGI has existed since 2022 (ChatGPT). There are now multiple companies in America and in China which have AGI-level agents. (It would be good to have a list of countries which are next in line to develop indigenous AGI capability.) Given that we are already in a state of coexistence between human and AGI, the next major transition is ASI, and that means, if not necessarily the end of humanity, the end of human control over human affairs. This most likely means that the dominant intelligence in the world is entirely nonhuman AI; it could also refer to some human-AI symbiosis, but again, it won't be natural humanity in charge. 

A world in which the development of AGI is allowed, let alone a world in which there is a race to create ever more powerful AI, is a world which by default is headed towards ASI and the dethronement of natural humanity. That is our world already, even if our tech and government leadership manage to not see it that way. 

So the most consequential thing that humans can care about right now is the transition to ASI. You can try to stop it from ever happening. or you can try to shape it in advance. Once it happens, it's over, humans per se have no further say, if they still exist they are at the mercy of the transhuman or posthuman agents now in charge. 

Now let me try to analyze your essay from this point of view. Your essay is meant as a critique of the paradigm according to which there is a race between America and China to create powerful AI, as if all that either side needs to care about is getting there first. Your message is that even if America gets to powerful AI (or safe powerful AI) first, the possibility of China (and in fact anyone else) developing the same capability would still remain. I see two main suggestions: a "mutual containment" treaty, in which both countries place bounds on their development of AI along with means of verifying that the bounds are being obeyed; and spreading around defensive measures, which make it harder for powerful AI to impose itself on the world. 

My take is that mutual containment really means a mutual commitment to stop the creation of ASI, a commitment which to be meaningful ultimately needs to be followed by everyone on Earth. It is a coherent position, but it's an uphill struggle since current trends are all in the other direction. On the other hand, I regard defense against ASI as impossible. Possibly there are meaningful defensive measures against lesser forms of AI, but ASI's relationship to human intelligence is like that of the best computer chess programs to the best human chess players - the latter simply have no chance in such a game. 

On the other hand, the only truly safe form of powerful AI is ASI governed by a value system which, if placed in complete control of the Earth, would still be something we could live with, or even something that is good for us. Anything less, e.g. a legal order in which powerful AI exists but there is a ban on further development, is unstable against further development to the ASI level. 

So there is a sense in which development of safe powerful AI really is all that matters, because it really has to mean safe ASI, and that is not something which will stay behind borders. If America, China, or any other country achieves ASI, that is success for everyone. But it does also imply the loss of sovereignty for natural humanity, in favor of the hypothetical benevolent superintelligent agent(s). 

It always seemed outlandish that in The Animatrix, the first AI city (01) was located in the Middle East... 

If we had limitless time, it would be interesting to know how this happened. I guess the prehistory of it involved Saudi Vision 2030 (e.g. the desert city Noem), and the general hypermodernization of Dubai. You can see precursors in the robot Sophia getting Saudi citizenship in 2017, and the UAE's "Falcon" LLM in 2023. 

But the initiative must have come from the American side - some intersection of the geopolitical brain trust around Trump, and the AI/crypto brain trust around David Sacks. The audacity of CEOs who can pick a country on the map and say, let's build a whole new technological facility here, combined with the audacity of grand strategists who can pick a region of the world and say, let's do a huge techno-economic deal with our allies here. 

There must have been some individual who first thought, hey, the Gulf Arabs have lots of money and electricity, that would be a good place to build all these AI data centers we're going to need; I wonder who it was. Maybe Ivanka Trump was telling Jared Kushner about Aschenbrenner's "Situational Awareness", they put 2 and 2 together, and it became part of the strategic mix along with "Trump Gaza number one", the new Syria, and whatever they're negotiating with Iran. 

Luckily I don’t think the Accelerationists have won control of the wheel

Could you expand on this? Also, have you had any interaction with accelerationists? In fact, are there any concrete Silicon Valley factions you would definitely count as accelerationists? 

The simpler explanation is that EY doesn't understand QM

Presumably you wouldn't say this of actual physicists who believe in MWI? 

I have tried looking at this from the perspective that we have had AGI since 2022 and ChatGPT. Creating ChatGPT didn't require an ecosystem, did it? Just a well-resourced nonprofit/startup with good researchers. 

So according to me, we've already had AGI for 2.5 years. We are still in a period of relative parity between humans and AI, in which we're still different enough, and it is still weak enough, that humans have the upper hand, and we're focused on exploring all possible variations on the theme of AI and human-AI relationship. 

The real question is when and how will it escape human control. That will be the real sign that we have "achieved" ASI. Will that result from an ecosystem and not a single firm? 

There seems to be an assumption that ASI will be achieved by continuing to scale up. All these analyses revolve around the economics of ever larger data centers and training runs, whether there's enough data and whether synthetic data is a good enough substitute, and so on. 

But surely that is just the dumbest way to achieve ASI. I'm sure some actors will keep pursuing that path. But we're also now in a world of aggressive experimentation with AI on every front. I have been saying that the birthplace of ASI will not be ever larger foundation models, but rather research swarms of existing and very-near-future AIs. Think of something that combines China's Absolute Zero Reasoner with the darwinism of Google AI's AlphaEvolve. Once they figure out how to implement John-von-Neumann-level intelligence in a box, I think we're done - and I don't think the economics of that requires an ecosystem, though it may be carried out by a Big Tech company that has an ecosystem (such as Google). 

This has never happened at OpenAI.

"This prompt (sometimes) makes ChatGPT think about terrorist organisations"

One is not philosophically obliged to regard the nature of reality as ineffable or inescapably uncertain. 

Quarks are a good place to explore this point. The human race once had no concept of quarks. Now it does. You say that inevitably, one day, we'll have some other concept. Maybe we will. But why is that inevitable? Why can't quarks just turn out to be part of how reality actually is? 

You cite Nagarjuna and talk about emptiness, so that gives me some idea of where you are coming from. This is a philosophy which emphasizes the role of concepts in constituting experience, and the role of the mind in constituting concepts, and typically concludes that reality has no essence, no nature that can be affirmed, because all such affirmations involve concepts that are introduced by the mind, rather than being inherent to anything. 

This conclusion I think is overreaching. I actually consider direct experience to be the ultimate proof that reality is not just formlessness carved by mind. Consciousness is not just raw being, it is filled with form. My words and concepts may not capture it properly, I may not even notice everything that is implied by what I see or what I am. But I do see that complexity and multiplicity are there in reality - at the very least, they are there in my own consciousness. 

Non-attachment to theories and concepts is a good thing if you're interested in truth, and know that you don't know the truth. It also has some pragmatic value if reality changes around you and you need to adapt. But in fundamental matters, one does not have to regard every concept and hypothesis that we have, as necessarily temporary. In some of them we may have latched onto the actual objective truth. 

P.S. Having criticized the philosophy of emptiness, let me add ironically that just a few hours ago, I investigated a proposal for AI alignment that someone had posted here a few months ago, and found it to be very good - and its model of the mind, a nondual viewpoint is the highest form. So your philosophy may actually put you in a good position to appreciate the nuances of this potentially important work. 

You could say there are two conflicting scenarios here: superintelligent AI taking over the world, and open-source AI taking over daily life. In the works that you mention, superintelligence comes so quickly that AI mostly remains a service offered by a few big companies, and open-source AI is just somewhere in the background. In an extreme opposite scenario, superintelligence might take so long to arrive, that the human race gets completely replaced by human-level AI before superintelligent AI ever exists. 

It would be healthy to have all kinds of combinations of these scenarios being explored. For example, you focus a bit on open-source AI as a bioterror risk. I don't think a supervirus is going to wipe out the human race or even end civilization, because (as Covid experience shows), we are capable of extreme measures in order to contain truly deadly disease. But a supervirus could certainly bring the world to a halt again, and if it was known to have been designed with open-source AI, that would surely have a huge impact on AI's trajectory. (I suspect that in such a scenario, AI for civilian purposes would suffer, but deep states worldwide would insist on pressing forward, and that there would also be a lobby arguing for AI as a defense against superviruses. Also, it's very plausible that a supervirus might be designed by AI, but that there would be no proof of it, in which case there wouldn't even be a backlash.) 

Another area where futurology about open-source AI might be good, is in the area of gradual disempowerment and replacement of humanity. We have societies with a division of roles, humans presently fill those roles but AI and robots will be capable of filling more and more of them; eventually every role in the economic, cultural, and political structure could be filled by AIs rather than by humans. The story of how that could happen, certainly deserves to be explored. 

Still another area when open-source AI scenarios deserve to be studied, is in the highly concrete realm of near-future economics and culture. What does an AI economy look like if o4-level models are just freely available? This really is an urgent question for anyone concerned with concrete questions like, who will lead the AI industry and how will it be structured, because there seem to be factions in both China and America who are thinking in this direction. One should want to understand what they envision, and what kind of competitive landscape they are likely to create in the short term. 

My own belief is that this would be such an upheaval, that it would inevitably end up invalidating many conventional political and economic premises. The current world order of billionaires and venture capitalists, stock markets and human democracies, I just don't see it surviving such a transition, even without superintelligence appearing. There are just too many explosive possibilities, too many new symbioses of AI with human mind, for the map of the world and the solar system to not be redrawn. 

However, in the end I believe in short timelines to superintelligence, and that makes all the above something of a secondary concern, because something is going to emerge that will overshadow humans and human-level AI equally. It's a little monotonous to keep referring back to Iain Banks's Culture universe, but it really is the outstanding depiction of a humanly tolerable world in which superintelligence has emerged. His starfaring society is really run by the "Minds", which are superintelligent AIs characteristically inhabiting giant spaceships or whole artificial worlds, and the societies over which they invisibly preside, include both biological intelligences (such as humans) and human-level AIs (e.g. the drones). The Culture is a highly permissive anarchy which mostly regulates itself via culture, i.e. shared values among human-level intelligences, but it has its own deep state, in the form of special agencies and the Minds behind them, who step in when there's a crisis that has escaped the Minds' preemptive strategic foresight. 

This is one model of what relations between superintelligence and lesser intelligences might be like. There are others. You could have an outcome in which there are no human-level intelligences at all, just one or more superintelligences. You could have superintelligences that have a far more utilitarian attitude to lower intelligences, creating them for temporary purposes and then retiring them when they are no longer needed. I'm sure there are other possibilities. 

The point is that from the perspective of a governing superintelligence, open-source AIs are just another form of lower intelligence, that may be useful or destabilizing depending on circumstance, and I would expect a superintelligence to decide how things should be on this front, and then to make it so, just as it would with every other aspect of the world that it cared about. The period in which open-source AI was governed only by corporate decisions, user communities, and human law would only be transitory. 

So if you're focused on superintelligence, the real question is whether open-source AI matters in the development of superintelligence. I think potentially it does - for example, open source is both a world of resources that Big Tech can tap into, as well as a source of destabilizing advances that Big Tech has to keep up with. But in the end, superintelligence - not just reasoning models, but models that reason and solve problems with strongly superhuman effectiveness - looks like something that is going to emerge in a context that is well-resourced and very focused on algorithmic progress. And by definition, it's not something that emerges incrementally and gets passed back and forth and perfected by the work of many independent hands. At best, that would describe a precursor of superintelligence. 

Superintelligence is necessarily based on some kind of incredibly powerful algorithm or architecture, that gets maximum leverage out of minimum information, and bootstraps its way to overwhelming advantage in all domains at high speed. To me, that doesn't sound like something invented by hobbyists or tinkerers or user communities. It's something that is created by highly focused teams of genius, using the most advanced tools, who are also a bit lucky in their initial assumptions and strategies. That is something you're going to find in an AI think tank, or a startup like Ilya Sutskever's, or a rich Big Tech company that has set aside serious resources for the creation of superintelligence. 

I recently posted that superintelligence is likely to emerge from the work of an "AI hive mind" or "research swarm" of reasoning models. Those could be open-source models, or they could be proprietary. What matters is that the human administrators of the research swarm (and ultimately, the AIs in the swarm itself) have access to their source code and their own specs and weights, so that they can engaged in informed self-modification. From a perspective that cares most about superintelligence, this is the main application of open source that matters. 

Somehow this has escaped comment, so I'll have a go. I write from the perspective of whether it's suitable as the value system of a superintelligence. If PRISM became the ethical operating system of a posthuman civilization born on Earth, for as long as that civilization managed to survive in the cosmos - would that be a satisfactory outcome? 

My immediate thoughts are: It has a robustness, due to its multi-perspective design, that gives it some plausibility. At the same time, it's not clear to me where the seven basis worldviews come from. Why those seven, and no others? Is there some argument that these seven form a necessary and sufficient basis for ethical behavior by human-like beings and their descendants? 

If I dig a little deeper into the paper, the justification is actually in part 2. Specifically, on page 12, six brain regions and their functions are singled out, as contributing to human decision-making at increasingly abstract levels (for the hierarchy, see page 15). The seven basis worldviews correspond to increasing levels of mastery of this hierarchy. 

I have to say I'm impressed. I figured that the choice of worldviews would just be a product of the author's intuition, but they are actually grounded in a theory of the brain. One of the old dreams associated with CEV, was that the decision procedure for a human-friendly AI would be extrapolated in a principled way from biological facts about human cognition, rather than just from a philosophical system, hallowed tradition, or set of community principles. June Ku's MetaEthical AI, for example, is an attempt to define an algorithm for doing this. Well, this is a paper written by a human being, but the principles in part 2 are sufficiently specific, that one could actually imagine an automated process following them, and producing a form of PRISM as its candidate for CEV! I'd like @Steven Byrnes to have a look at this.  

We had Golden Gate Claude, now we have White Genocide Grok... 

This seems like a Chinese model for superintelligence! (All the authors are Chinese, though a few are working in the West.) Not in the AIXI sense of something which is optimal from the beginning, but rather something that could bootstrap its way to superintelligence. One could compare it to Schmidhuber's Godel machine concept, but more concrete, and native to the deep learning era. 

(If anyone has an argument as to why this isn't a model that can become arbitrarily intelligent, I'm interested.) 

If I have understood correctly, you're saying that OpenAI should be forecasting greater revenue than this, if they truly think they will have AIs capable of replacing entire industries. But maybe they're just being cautious in their forecasts? 

Suppose I have a 3d printing / nanotechnology company, and I think that a year from now I'll have an unlimited supply of infinity boxes capable of making any material artefact. World manufacturing is worth over US$10 trillion. If I thought I could put it all out of business, by charging just 10% of what current manufacturers charge, I could claim expected revenue of $1 trillion. 

Such a prediction would certainly be attention-grabbing, but maybe it would be reckless to make it? Maybe my technology won't be ready. Maybe my products will be blocked from most markets. Maybe someone will reverse-engineer and open-source the infinity boxes, and prices will crash to $0. Maybe I don't want the competition or the government to grasp just how big my plans are. Maybe the investors I want wouldn't believe such a scenario. There are a lot of reasons why a company that thinks it might be able to take over the economy or even the world, would nonetheless not put that in its prospectus. 

OpenAI gets a lot of critical attention here, because it's been the leader in many ways. But what about Google AI? Many people think it's the leader now, yet I don't see anywhere near as much critical scrutiny of its decision-making process. 

That gives "Eternal September" a new meaning... 

The view that Heisenberg advocates - reductionism had reached a limit, and a new paradigm was needed - was a highly influential school of thought in the 1960s. In particle physics, there is a mathematical object called the S-matrix (scattering matrix), which tabulates scattering amplitudes (the quasiprobability that if these N particles enter a collision, these other M particles will be what comes out). Quantum electrodynamics (a theory with electrons and photons, let's say) is a prototypical quantum field theory in which the S-matrix can be calculated from the stipulation that electrons and photons are fundamental. For the weak interactions (later unified with electromagnetism), this reductionist method also works. 

But for the strong interactions, field theory looked intractable, and a new philosophy was advanced that the S-matrix itself should be the central mathematical object in the theory. Remember that quarks were never seen by themselves, only protons, neutrons, and a hundred other types of "hadron". The idea of nuclear democracy was that the S-matrix for these hundred seemingly equi-fundamental particle species, would be derived from postulates about the properties of the S-matrix, rather than from an underlying field theory. This was called the bootstrap program, it is how the basic formulae of string theory were discovered (before they had even been identified as arising from strings), and it's still used to study the S-matrix of computationally intractable theories. 

These days, the philosophy that the S-matrix is primary, still has some credibility in quantum gravity. Here the problem is not that we can't identify ultimate constituents, but rather that the very idea of points of space-time seems problematic, because of quantum fluctuations in the metric. The counterpart of the 1960s skepticism about quarks, would be that the holographic boundary of space-time is fundamental. For example, in the AdS/CFT correspondence, scattering events in Anti de Sitter space (in which particles approach each other "from the boundary", interact, and then head back to the boundary) can be calculated entirely within the boundary CFT, without any reference to AdS space at all, which is regarded as emergent from the boundary space. The research program of celestial holography is an attempt to develop the same perspective within the physically relevant case of flat space-time. The whole universe that we see, would be a hologram built nonlocally from entanglement within a lower-dimensional space... 

The eventual validation of quarks as particles might seem like a sign that this radical version of the holographic philosophy will also be wrong in the end, and perhaps it will be. But it really shows the extent to which the late thoughts of Heisenberg are still relevant. Holographic boundaries are the new S-matrix, they are a construct which has made quantum gravity uniquely tractable, and it's reasonable to ask if they should be treated as fundamental, just as it was indeed entirely reasonable for Heisenberg and the other S-matrix theorists to ask whether the S-matrix itself is the final word. 

Ugh, I was using LW's custom reaction emoticons to annotate this comment, and through a fumble, have ended up expressing a confidence of 75% in the scenario that AI will "otherwise enforce on us a pause", and I don't see how to remove that annotation. 

I will say that, alignment aside, the idea that an advanced AI will try to halt humanity's AI research so it doesn't produce a rival, makes a lot of sense to me. 

Ted Cruz mentioned how his daughter was using ChatGPT when texting him. I wonder how many of these senators and CEOs, and their staffers and advisors, are already doing the same, when they try to decide AI policy. I guess that would be an example of weak-to-strong superalignment :-) 

Heisenberg versus quarks is one of the best lifelong physics thinkers, encountering one of the most subtle topics in physics. When he says

The questions about the statistics of quarks, about the forces that hold them together, about the particles corresponding to these forces, about the reasons why quarks never appear as free particles, about the pair-creation of quarks in the interior of the elementary particle -- all these questions are more or less left in obscurity.

... he is raising all the right questions, and their resolution required a field theory with completely unprecedented behaviors. The statistics of quarks required a new threefold quantum property, "color"; the forces that hold them together were described by a new kind of quantum field, a strongly coupled Yang-Mills field; the particles corresponding to those forces are the gluons, and like quarks, they never appear as free particles, because of a new behavior, "confinement". 

The deeper story here is a struggle of paradigms in the 1960s, between the old search for the most elementary particles, and the idea of a "nuclear democracy" of transformations among a large number of equally fundamental particle species. We now see those many particle types as due to different combinations of quarks, but at the time no-one understood how quarks could literally be particles (for the reasons touched on by Heisenberg), and they were instead treated as bookkeeping devices, akin to conserved quantities. 

The quark theorists won, once the gauge theory of gluons was figured out; but nuclear democracy (in the guise of "S-matrix theory", pioneered by Heisenberg) left its mark too, because it gave rise to string theory. They aren't even completely distinct paradigms; there are very close relationships between gauge theory and string theory, though not close enough that we know exactly what string theory corresponds to the quarks and gluons of the standard model. 

Incidentally, is there any meaningful sense in which we can say how many "person-years of thought" LLMs have already done? 

We know they can do things in seconds that would take a human minutes. Does that mean those real-time seconds count as "human-minutes" of thought? Etc. 

Liron: ... Turns out the answer to the symbol grounding problem is like you have a couple high dimensional vectors and their cosine similarity or whatever is the nature of meaning.

Could someone state this more clearly?

Jim: ... a paper that looked at the values in one of the LLMs as inferred from prompts setting up things like trolley problems, and found first of all, that they did look like a utility function, second of all, that they got closer to following the VNM axioms as the network got bigger. And third of all, that the utility function that they seemed to represent was absolutely bonkers 

What paper was this? 

scientifically falsifiable

How is it falsifiable?

I think it's very good to have people around who are saying "cut back on social media", "get off social media", as a counter to its addictive power. 

And yet... If I take the title literally, I am being told that I should quit social media entirely, as soon as possible, because in the near future, it will be so addictive that I will be literally unable to quit. 

When you first raised this idea, I asked what will happen to people who don't get out in time? In this post, we now have a concrete scenario. The protagonist doesn't die. They don't go mad. They don't become anyone's minion. They just... spend a lot of time irritated, spend all day watching videos, and lose touch with some real people. 

Well, that's nobody's ideal, but it's not actually worse than the human condition has been, for large numbers of people throughout history. "Lives of quiet desperation" I think have been pretty common in the agricultural and industrial eras. In terms of historical experience, it is actually normal for people to end up limping through life with some affliction that they never quite get over, whether it's injury, trauma, some familial or national destiny that they just can't escape... To learn that, in the information age, some people become unwholesome computer or media addicts, is just to write the next chapter of that. 

Let me be clear, I'm not quite urging apathy about social media addiction. It's just that I was expecting something more apocalyptic as the payoff, that humanity would be utter captives of the content farms, perhaps later to be herded into Matrix pods or assembled into armies of meme-controlled zombies. Instead, what you're describing is more like a chronic misery specific to the information age. 

It's like someone warning that if you abandon hunting and gathering, you'll end up standing around all day watching farm animals, or if you leave the farm for the big industrial city, you'll end up stooped and maimed by factory work. All that actually happened, but there were also huge upsides to the new order in each case. 

After all, there's actually a lot of good stuff that comes through social media. With a keyword search, I can find up-to-the-second information and perspectives, on something that is happening, including censored perspectives. I can follow the news about topics that interest only a very niche audience. I can eavesdrop on, and even participate in, all kinds of discussions that would otherwise be out of my reach. I can track down lost friends, find work, simply indulge my curiosity. 

Of course there are formidable downsides too. You can overindulge (I have a weakness for reaction videos), you can burn out certain faculties, you can forget your own life amidst a million distractions, and just as in real life, there are far worse things lying in wait: scammers, grifters, toxic personalities and communities, political botnets; and perhaps there are even architects of addiction who deserve death as much as any fentanyl dealer does. 

It's just that you haven't really made the case, that the social Internet will become nothing but a prison of blighted lives. Life in that space is much more like living in a city. There are risks and evils specific to urban life, and city dwellers must learn to avoid them, and lots of people fall prey to them. But there are also good things that can only happen in cities, and there are new good things that only happen online. 

I haven't mentioned the AI factor so far, but it is central to your scenario. My response again is that there are positives and negatives, and in some cases it may not even be clear which is which, The combination of AI and social media may lead to new symbioses that look horrifying to outsiders, but which have a merit and integrity of their own. As AI becomes more and more capable, the question of AI on social media just blends into the broader question of humanity's destiny in a world with AI, and ultimately, a world with artificial superintelligence. 

How often will a civilization with the capability to perform such a simulation, have anything to learn from it? 

Yes, thanks. And someone should do the same analysis, regarding coverage of AI 2027 in American/Western media. (edit: A quick survey by o3)

spreading the idea of "heroic responsibility" seems, well, irresponsible

Is this analogous to saying "capabilities research is dangerous and should not be pursued", but for the human psyche rather than for AI?

Your comment has made me think rather hard on the nature of China and America. The two countries definitely have different political philosophies. On the question of how to avoid dictatorship, you could say that the American system relies on representation of the individual via the vote, whereas the Chinese system relies on representation of the masses via the party. If an American leader becomes an unpopular dictator, American individuals will vote them out; if a Chinese leader becomes an unpopular dictator, the Chinese masses will force the party back on track. 

Even before these modern political philosophies, the old world recognized that popular discontent could be justified. That's the other side of the mandate of heaven: when a ruler is oppressive, the mandate is withdrawn, and revolt is justified. Power in the world of monarchs and emperors was not just about who's the better killer; there was a moral dimension, just as democratic elections are not just a matter of who has the most donors and the best public relations. 

Returning to the present and going into more detail, America is, let's say, a constitutional democratic republic in which a party system emerged. There's a tension between the democratic aspect (will of the people) and the republican aspect (rights of the individual), which crystallized into an opposition found in the very names of the two main parties; though in the Obama-Trump era, the ideologies of the two parties evolved to transnational progressivism and populist nationalism. 

These two ideologies had a different attitude to the unipolar world-system that America acquired, first by inheriting the oceans from the British empire, and then by outlasting the Russian communist alternative to liberal democracy, in the ideological Cold War. For about two decades, the world system was one of negotiated capitalist trade among sovereign nations, with America as the "world police" and also a promoter of universal democracy. In the 2010s, this broke down as progressivism took over American institutions, including its external relations, and world regions outside the West increasingly asserted their independence of American values. The appearance of populist nationalism inside America makes sense as a reaction to this situation, and in the 2020s we're seeing how that ideology acts within the world system: America is conceived as the strongest great power, acting primarily in the national interest, with a nature and a heritage that it will not try to universalize. 

So that's our world now. Europe and its offshoots conquered the world, but imperialism was replaced by nationalism, and we got the United Nations world of several great powers and several hundred nations. America is the strongest, but the other great powers are now following their own values, and the strongest among the others is China. America is a young offspring of Europe on a separate continent, modern China is the latest iteration of civilization on its ancient territory. The American political philosophy is an evolution of some ancient European ideas; the Chinese political philosophy is an indigenous adaptation of an anti-systemic philosophy from modern Europe. 

One thing about Chinese Marxism that is different from the old Russian Marxism, is that it is more "voluntarist". Mao regarded Russian Marxism as too mechanical in its understanding of history; according to Mao, the will of the people and the choices of their political leadership can make a difference to events. I see an echo of this in the way that every new secretary of the Chinese Communist Party has to bring some new contribution to Marxist thought, most recently "Xi Jinping Thought". The party leader also has to be the foremost "thought leader" in Chinese Marxism, or they must at least lend their name to the ideological state of the art (Wang Huning is widely regarded as the main Chinese ideologist of the present). This helps me to understand the relationship between the party and the state institutions. The institutions manage society and have concrete responsibilities, while the party determines and enforces the politically correct philosophy (analogous to the role that some now assign to the Ivy League universities in America). 

I've written all this to explain in greater detail, the thinking which I believe actually governs China. To just call China an oppressive dictatorship, is to miss the actual logic of its politics. There are certainly challenges to its ideology. For example, the communist ideology was originally meant to ensure that the country was governed in the interest of the worker and peasant classes. But with the tolerance of private enterprise, more and more people become the kind of calculating individual agent you have under capitalism, and arguably representative democracy is more suited to such a society. 

One political scientist argues that ardor for revolutionary values died with Mao, leaving a void which is filled partly by traditional values and partly by liberal values. Perhaps it's analogous to how America's current parties and their ideologies are competing for control of a system that (at least since FDR) was built around liberal values; except that in China, instead of separate parties, you have factions within the CCP. In any case, China hasn't tilted to Falun Gong traditionalism or State Department democratization, instead Xi Jinping Thought has reasserted the centrality of the party to Chinese stability and progress. 

Again, I'm writing this so we can have a slightly more concrete discussion of China. There's also a bunch of minor details in your account that I believe are wrong. For example, "Nationalist China" (the political order on the Chinese mainland, between the last dynasty and the communist victory) did not have regular elections as far as I know. They got a parliament together at the very beginning, and then that parliament remained unchanged until they retreated to Taiwan (they were busy with famines, warlordism, and the Japanese invasion); and then Taiwan remained a miitary-run regime for forty years. The Uighurs are far from being the only significant ethnic group apart from the Han, there are several others of the same size. Zhang Yiming and Rubo Liang are executives from Bytedance, the parent company of Tiktok (consider the relationship between Google/Alphabet and YouTube); I think Zhang is the richest man in China, incidentally. 

I could also do more to explain Chinese actions that westerners find objectionable, or dig up the "necessary evils" that the West itself carries out. But then we'd be here all day. I think I do agree that American society is politically friendlier to the individual than Chinese society; and also that American culture, in its vast complexity, contains many many valuable things (among which I would count, not just notions like rule of law, human rights, and various forms of respect for human subjectivity, but also the very existence of futurist and transhumanist subcultures; they may not be part of the mainstream, but it's significant that they get to exist at all). 

But I wanted to emphasize that China is not just some arbitrary tyranny. It has its freedoms, it has its own checks and balances, it has its own geopolitical coalitions (e.g. BRICS) united by a desire to flourish without American dependence or intervention. It's not a hermit kingdom that tunes out the world (witness, for example, the frequency with which very western-sounding attitudes emerge from their AIs, because of the training data that they have used). If superintelligence does first emerge within China's political and cultural matrix, it has a chance of being human-friendly; it will just have arrived at that attractor from a different starting point, compared to the West. 

Some of the recent growing pains of AI (flattery, selfish rule-breaking) seem to be reinventing aspects of human nature that we aren't proud of, but which are ubiquitous. It's actually very logical that if AIs are going to inhabit more and more of the social fabric, they will manifest the full spectrum of social behaviors. 

OpenAI in particular seems to be trying to figure out personality, e.g. they have a model called "Monday" that's like a cynical comedian that mocks the user. I wonder if the history of a company like character.ai, whose main product is AI personality, can help us predict where OpenAI will take this. 

I can imagine an argument analogous to Eliezer's old graphic illustrating that it's a mistake to think of a superintelligence as Einstein in a box. I'm referring to the graphic where you have a line running from left to right, on the left you have chimp, ordinary person, Einstein all clustered together, and then far away on the other side, "superintelligence", the point being that superintelligence far transcends all three. 

In the same way, the nature of the world when you have a power that great is so different that the differences among all human political systems diminish to almost nothing by comparison, they are just trivial reorderings of power relations among beings so puny as to be almost powerless. Neither the Chinese nor the American system is built to include intelligent agents with the power of a god, that's "out of distribution" for both the Communist Manifesto and the Federalist Papers. 

Because of that, I find it genuinely difficult to infer from the nature of the political system, what the likely character of a superintelligence interested in humanity could be. I feel like contingencies of culture and individual psychology could end up being more important. So long as you have elements of humaneness and philosophical reflection in a culture, maybe you have a chance of human-friendly superintelligence emerging.  

Gold was currency, and is still used as a hedge against fiat currency. 

the suspicion [...] that there hasn't been much material growth in the world over the last 40 or so years compared to before

I assume most of that growth occurred in China. 

Has GDP essentially been goodharted by central banks in recent times?

What can central banks do to affect GDP growth? 

小句子很好。为什么？清晰。我的句子又长又抽象。不清楚。现在简短清晰了。

很好。思路清晰。为什么？简单需要深刻。

There are a number of aspects that make fire alarms less likely in the AI 2027 scenarios compared to what I consider likely - e.g. having 2 projects that matter, whereas I expect more like 3 to 6 such projects.

I agree about the plurality of projects. AI 2027 has an American national project and a Chinese national project, whereas at present both countries have multiple companies competing with each other. 

AI 2027 also has the two national AIs do a secret deal with each other. My own thought about superintelligence does treat it as a winner-take-all race, so "deals" don't have the same meaning as in situations where the parties  actually have something to offer each other. There's really only room for pledges or promises, of the form "If I achieve superintelligence first, I promise that I will use my power in the service of these goals or values". 

So my own model of the future has been that there will be more than two contenders, and that one of them will cross the threshold of superintelligence first and become the ultimate power on Earth. At that point it won't need anyone or anything else; all other ex-contenders will simply be at the winner's mercy. 

A full-fledged arms race would involve the racers acting as if the second place finisher suffers total defeat.

I don't get the impression that AI companies currently act like that. They seem to act like first place is worth trillions of dollars, but also like employees at second and third place "finishers" will each likely get something like a billion dollars.

Even before AI, it was my impression of big tech companies like Microsoft, Amazon, Google, that they are quite willing to form cooperative relationships, but they also have no compunction about attempting to become dominant in every area that they can. If any of them could become an ultimate all-pervasive monopoly, they would. Is there anything in the behavior of the AI companies that looks different? 

I take this to mostly be a response to the idea that humanity will be protected by decentralization of AI power, the idea apparently being that your personal AI or your society's AIs will defend you against other AIs if that is ever necessary. 

And what I think you've highlighted, is that this is no good if your defensive AIs are misaligned (in the sense of not being properly human-friendly or even just "you"-friendly), because what they will be defending are their misaligned values and goals. 

As usual, I presume that the AIs become superintelligent, and that the situation evolves to the point that the defensive AIs are in charge of the defense from top to bottom. It's not like running an antivirus program, it's like putting a new emergency leadership in charge of your entire national life. 

Maybe OpenAI did something to prevent its AIs from being pro-Hamas, in order to keep the Trump administration at bay, but it was too crude a patch and now it's being triggered at inappropriate times. 

Old-timers might remember that we used to call lying, "hallucination". 

Which is to say, this is the return of a familiar problem. GPT-4 in its early days made things up constantly, that never completely went away, and now it's back. 

Did OpenAI release o3 like this, in order to keep up with Gemini 2.5? How much does Gemini 2.5 hallucinate? How about Sonnet 3.7? (I wasn't aware that current Claude has a hallucination problem.)

We're supposed to be in a brave new world of reasoning models. I thought the whole point of reasoning was to keep the models even more based in reality. But apparently it's actually making them more "agentic", at the price of renewed hallucination? 

Is there a name for the phenomenon of increased intelligence or increased awareness leading to increased selfishness? It sounds like something that a psychologist would have named. 

The four questions you ask are excellent, since they get away from general differences of culture or political system, and address the processes that are actually producing Chinese AI. 

The best reference I have so far is a May 2024 report from Concordia AI on "The State of AI Safety in China". I haven't even gone through it yet, but let me reproduce the executive summary here: 

The relevance and quality of Chinese technical research for frontier AI safety has increased substantially, with growing work on frontier issues such as LLM unlearning, misuse risks of AI in biology and chemistry, and evaluating "power-seeking" and "self-awareness" risks of LLMs. 

There have been nearly 15 Chinese technical papers on frontier AI safety per month on average over the past 6 months. The report identifies 11 key research groups who have written a substantial portion of these papers. 

China’s decision to sign the Bletchley Declaration, issue a joint statement on AI governance with France, and pursue an intergovernmental AI dialogue with the US indicates a growing convergence of views on AI safety among major powers compared to early 2023. 

Since 2022, 8 Track 1.5 or 2 dialogues focused on AI have taken place between China and Western countries, with 2 focused on frontier AI safety and governance. 

Chinese national policy and leadership show growing interest in developing large models while balancing risk prevention. 

Unofficial expert drafts of China’s forthcoming national AI law contain provisions on AI safety, such as specialized oversight for foundation models and stipulating value alignment of AGI. 

Local governments in China’s 3 biggest AI hubs have issued policies on AGI or large models, primarily aimed at accelerating development while also including provisions on topics such as international cooperation, ethics, and testing and evaluation. 

Several influential industry associations established projects or committees to research AI safety and security problems, but their focus is primarily on content and data security rather than frontier AI safety. 

In recent months, Chinese experts have discussed several focused AI safety topics, including “red lines” that AI must not cross to avoid “existential risks,” minimum funding levels for AI safety research, and AI’s impact on biosecurity.

So clearly there is a discourse about AI safety there, that does sometimes extend even as far as the risk of extinction. It's nowhere near as prominent or dramatic as it has been in the USA, but it's there.