Posts
Comments
I expected it to become rarer
Only a vanishingly small number of people sign up for cryonics - I think it would be just a few thousand people, out of the entirety of humanity. Even among Less Wrong rationalists, it's never been that common or prominent a topic I think? - perhaps because most of them are relatively young, so death feels far away.
Overall, cryonics, like radical life extension in general, is one of the many possibilities of existence that the human race has neglected via indifference. It's popular as a science fiction theme but very few people choose to live it in reality.
Because I think the self is possibly based on quantum entanglement among neurons, I am personally skeptical of certain cryonic paradigms, especially those based on digital reconstruction rather than physical reanimation. Nonetheless, I think that in a sane society with a developed economy, cryonic suspension would be a common and normal thing by now. Instead we have our insane and tragic world where people are so beaten down by life that, e.g. the idea of making radical rejuvenation a national health research priority sounds like complete fantasy.
I sometimes blame myself as part of the problem, in that I knew about cryonics, transhumanism, etc., 35 years ago. And I had skills, I can write, I can speak in front of a crowd - yet what difference did I ever make? I did try a few times, but whether it's because I was underresourced, drawn to too many other purposes at once, insufficiently machiavellian for the real world of backstabbing competition, or because the psychological inertia of collective indifference is genuinely hard to move, I didn't even graduate to the world of pundit-influencers with books and websites and social media followers. Instead I'm just one more name in a few forum comment sections.
Nonetheless, the human race has in the 2020s stumbled its way to a new era of technological promise, to the point that just an hour ago, the world's richest man was telling us all, on the social network that he owns, that he plans to have his AI-powered humanoid robots accompanying human expeditions to Mars a few years from now. And more broadly speaking, AI-driven cures for everything are part of the official sales pitch for AI now, along with rapid scientific and technological progress on every front, and leisure and self-actualization for all.
So even if I personally feel left out and my potential contributions wasted, objectively, the prospects of success for cryonics and life extension and other such dreams is probably better than it's ever been - except for that little worry that "the future doesn't need us", and that AI might develop an agenda of its own that's orthogonal to the needs of the human race.
After many failed tries, I got it down to 5%. But it wasn't a method that would be useful in the real world :-(
I asked because I'm talking with you and I wanted to know *your* reasoning as to why a technical solution to the alignment of superintelligence is impossible. It seems to be "lots of people see lots of challenges and they are too many to overcome, take it up with them".
But it's just a hard problem, and the foundations are not utterly mysterious. Humanity understands quite a lot about the physical and computational nature of our reality by now.
Maybe it would be more constructive to ask how you envisage achieving the political impossible of stopping the worldwide AI race, since that's something that you do advocate.
Is there someone you regard as the authority on why it can't be done? (Yudkowsky? Yampolskiy?)
Because what I see, are not problems that we know to be unsolvable, but rather problems that the human race is not seriously trying to solve.
So what's the impossible thing - identifying an adequate set of values? instilling them in a superintelligence?
The denial of a self has long seemed to me a kind of delusion. I am very clearly having a particular stream of consciousness. It's not an arbitrary abstraction to say that it includes some experiences and does not include others. To say there is a self, is just to say that there is a being experiencing that stream of consciousness. Would you deny that, or are you saying something else?
Do you perceive the irony in telling me my hope has "zero percent chance" of happening, then turning around and telling me to do the impossible? I guess some impossibles are more impossible than others.
In fact I've spent 30 years attempting various forms of "the impossible" (i.e. things of unknown difficulty that aren't getting done), it's kind of why I'm chronically unemployed and rarely have more than $2000 in the bank. I know how to be audaciously ambitious in unpromising circumstances and I know how to be stubborn about it.
You like to emphasize the contingency of history as a source of hope. Fine. Let me tell you that the same applies to the world of intellectual discovery, which I know a lot better than I know the world of politics. Revolutionary advances in understanding can and do occur, and sometimes on the basis of very simple but potent insights.
I'm going to start keeping track of opaque or subtle outsiders who aren't very formal but who might have a piece of the ultimate alignment puzzle. At the very least, these are posts which an alignment theorist (whether AI or human) ought to be able to say something about, e.g. if there is a fallacy or omission, they should be able to identify it.
Along with this post, I'll mention
the intent behind your question
I guess I'm expressing doubt about the viability of wise or cautious AI strategies, given our new e/acc world order, in which everyone who can, is sprinting uninhibitedly towards superintelligence.
My own best hope at this point is that someone will actually solve the "civilizational superalignment" problem of CEV, i.e. learning how to imbue autonomous AI with the full set of values (whatever they are) required to "govern" a transhuman civilization in a way that follows from the best in humanity, etc. - and that this solution will be taken into account by whoever actually wins the race to superintelligence.
This is a strategy that can be pursued within the technical communities that are actually designing and building frontier AI. But the public discussion of AI is currently hopeless, it's deeply unrealistic about the most obvious consequences of creating new intelligences that are smarter, faster, and more knowledgeable than any human, namely, that they are going to replace human beings as masters of the Earth, and eventually that they will simply replace the entire human race.
What would their actual platform be?
Victory is the aim of war, not suffering.
Every so often I'll declare something to be the state of the art in AI-takeover short fiction. It was @gwern's Clippy in 2022 and @Gabe M's "Scale Was All We Needed, At First" in 2024. I think we have a new leader.
the pop-culture chakra model comes from modern Theosophical books, not ancient Indian scripture
Kind of true, although the core of it is a system of 6 or 7 chakras which the author acknowledges (way down in the comments) was dominant in India "by 1500".
This made me curious about the credentials of Transcendental Meditation, the system espoused by David Lynch (RIP) and Jerry Seinfeld among others. Turns out their guru was a student of the head of one of the four big Vedic monasteries founded over 1000 years ago by the Kant of Hinduism (that's just my name for him), Adi Shankara. So at least in this case, there are no troubling Russian or British intermediaries. :-)
I can't bring myself to read it properly. The author has an ax to grind, he wants interplanetary civilization and technological progress for humanity, and it's inconvenient to that vision if progress in one form of technology (AI) has the natural consequence of replacing humanity, or at the very least removing it from the driver's seat. So he simply declares "There is No Reason to Think Superintelligence is Coming Soon", and the one doomer strategy he does approve of - the enhancement of human biological intelligence - happens to be one that once again involves promoting a form of technological progress.
If there is a significant single failure behind getting to where we are now, perhaps it is the dissociation between "progress in AI" and "humanity being surpassed and replaced by AI" that has occurred. It should be common sense that the latter is the natural outcome of creating superhuman AI.
Yes, this is nothing like e/acc arguments. e/acc don't argue in favour of AI takeover; they refuse to even think about AI takeover. e/acc was "we need more AI for everyone now, or else the EAs will trap us all in stagnant woke dystopia indefinitely". Now it's "American AI must win or China will trap us in totalitarian communist dystopia indefinitely".
if you get enough meditative insight you'll transcend the concept of a self
What is the notion of self that you transcend, what does it mean to transcend it, and how does meditation cause this to happen?
Is there some way to use LLMs to efficiently simulate different kinds of AI futures, including extremely posthuman scenarios? I mean "simulation" in a primarily literary sense - via fictional vignettes and what-if essays - though if that can usefully be supplemented with other tools, all the better.
But this isn't an all-or-nothing choice. If you hurt your fingers getting them caught in a door, you suffer, but you don't want to die because of it, do you? Any ideas on where to draw the line?
Knowledge is power, and superintelligence means access to scientific and engineering knowledge at a new level. Your analysis seems to overlook this explanatory factor. We expect that superintelligence generically grants access to a level of technological capability, that includes engineering on astronomical scales; the ability to read, copy, and modify human minds, as well as simply make humanlike minds with arbitrary dispositions; and a transformative control over material things (including living beings) whose limits are hard to identify. In other words, any superintelligence should have a capacity to transform the world with "godlike" power and precision. This is why the advent of superintelligence, whether good, bad, or weird in its values, has this apocalyptic undercurrent in all scenarios.
In "The Autodidactic Universe", the authors try to import concepts from machine learning into physics. In particular, they want to construct physical models in which "the Universe learns its own physical laws". In my opinion they're not very successful, but one might wish to see whether their physicalized ML concepts can be put to work back in ML, in the context of a program like yours.
This comment has been on my mind a lot the past week - not because I'm not ambitious, but because I've always been ambitious (intellectually at least) and frustrated in my ambitions. I've always had goals that I thought were important and neglected, I always directly pursued them from a socially marginal position rather than trying to make money first (or whatever people do when they put off their real ambitions), but I can't say I ever had a decisive breakthrough, certainly not to recognition. So I only have partial progress on a scattered smorgasbord of unfulfilled agendas, and meanwhile, after OpenAI's "o3 Christmas" and the imminent inauguration of an e/acc administration in the USA, it looks more than ever that we are out of time. I would be deeply unsurprised if it's all over by the end of the year.
I'm left with choices like (1) concentrate on family in the final months (2) patch together what I have and use AI to quickly make the best of it (3) throw myself into AI safety. In practice they overlap, I'm doing all three, but there are tensions between them, and I feel the frustration of being badly positioned while also thinking I have no time for the meta-task of improving my position.
You first might want to distinguish between national AI projects that are just about boosting the AI economy or managing the use of AI within government, and government-backed research which is specifically aimed at the AGI frontier. Presumably it's the latter that you're talking about.
There is also the question of what a government would think it was doing, in embarking on such a project. The commercial enterprise of creating AI is already haunted by the idea that it would be bad for business if your creation wiped out the human race. That hasn't stopped anyone, but the fear is there, overcome only by greed.
Now, what about politicians and public servants, generals and spymasters? How would they feel about leading a race to create AI? What would they think they were doing? Creating artificial super-scientists, super-soldiers, super-strategists? Compared to Silcon Valley, these people are more about the power motive than the profit motive. What, apart from the arms race, do they have to lure them along the AI path, comparable to the dream of uber-wealth that drives the tech oligarchs? (In dictatorships, I suppose there is also the dream of absolute personal power to motivate them.)
Apart from the arms race, the vision that seems to animate pro-AI western elites, is economic and strategic competition among nations. If China takes the lead in AI, it will have the best products and the best technologies and it will conquer the world that way. So I guess the thinking of Trump 2.0's AI czar David Sacks (a friend of Thiel and Musk), and the people around him, is going to be some mixture of these themes - the US must lead because AI is the key to economic, technological, and military superiority in the 21st century.
Now I think that even the most self-confident, gung-ho, born-to-rule man-of-destiny who gets involved in the AI race, is surely going to have a moment when they think, am I just creating my own replacement here? Can even my intellect, and my charisma, and my billions, and my social capital, really compete with something smarter than me, and a thousand times faster than me, and capable of putting any kind of human face on its activities?
I'm not saying they're going to have a come-to-Yudkowsky moment and realize, holy crap, we'd better shut this down after all. Their Darwinist instincts will tell them that if they don't create AI first, someone else will. But perhaps they will want to be reassured. And this may be one area where techies similar to Ilya Sutskever, and Yann Lecun, and Alec Radford - i.e. the technical leads in these frontier AI research programs - may have a role in addition to their official role as chief of R&D.
The technical people have their own dreams about what a world of AGI and ASI could look like too. They may have a story about prosperity and human flourishing with AI friends and partners. Or maybe they have a story just for their CEO masters, that even the most powerful AI, if properly trained, will just be 100% an extension of their own existing will. And who knows what kind of transhuman dreams they entertain privately, as well?
These days, there's even the possibility that the AI itself is whispering to the corporate, political, and military leadership, telling them what they want to hear...
I am very much speculating here, I have no personal experience of, or access to, these highest levels of power. But the psychology and ideology of the "decision-makers" - who really just seem to be riding the tiger of technical progress at this point - is surely an important feature of any such AGI Manhattan Project, too.
Regarding Musk and Thiel, foremost they are billionaire capitalists, individuals who built enormous business empires. Even if we assume your thinking about the future is correct, we shouldn't assume that they have reproduced every step of it. You may simply be more advanced in your thinking about the future than they are. Their thought about the future crystallized in the 1980s, when they were young. Since then they have been preoccupied with building their empires.
This raises the question, how do they see the future, and their relationship to it? I think Musk's life purpose is the colonization of Mars, so that humanity's fate isn't tied to what happens on Earth. Everything else is subordinate to that, and even robots and AI are just servants and companions for humanity in its quest for other worlds. As for Thiel, I have less sense of the gestalt of his business activities, but philosophically, the culture war seems very important to him. He may have a European sense of how self-absorbed cultural elites can narrow a nation's horizons, that drives his sponsorship of "heterodox" intellectuals outside the academy.
If I'm right, the core of Musk's futurism is space colonization, and the core of Thiel's futurism is preserving an open society. They don't have the idea of an intelligence singularity whose outcome determines everything afterwards. In this regard, they're closer to e/acc than singularity thinking, because e/acc believes in a future that always remains open, uncertain, and pluralist, whereas singularity thinking tends towards a single apocalyptic moment in which superintelligence is achieved and irreversibly shapes the world.
There are other reasons I can see why they would involve themselves in the culture war. They don't want a socialism that would interfere with their empires; they think (or may have thought until the last few years) that superintelligence is decades away; they see their culture war opponents as a threat to a free future (whether that is seen in e/acc or singularity terms), or even to the very existence of any kind of technological future society.
But if I were to reduce it to one thing: they don't believe in models of the future according to which you get one thing right and then utopia follows, and they believe such thinking actually leads to totalitarian outcomes (where their definition of totalitarian may be, a techno-political order capable of preventing the building of a personal empire). Musk started OpenAI so Google wouldn't be the sole AI superpower; he was worried about centralization as such, not about whether they would get the value system right. Thiel gave up on MIRI's version of AI futurology years ago as a salvationist cult; I think he would actually prefer no AI to aligned AI, if the latter means alignment with a particular value system rather than alignment with what the user wants.
Thank you for the answer! What model was used - 4o? o1?
Was this idea or post AI-assisted in any way? If it was, it seems a cut above the usual AI-generated material, and I would be interested to know your methods.
Is there a good Russian-language introduction to AI alignment?
I think we could certainly afford to have much more discussion of this topic. The two forms of takeover are not absolutely distinct. Any humans who take over the world are going to be AI-enhanced, and all their observing, deciding, and acting will be heavily AI-mediated. On the other hand, any AI that takes over the world will be the product of human design and human training, and will start out embedded in an organization of human beings.
Ideally, people would actually "solve ethics" and how to implement it in an AI, and we would only set superintelligence in motion having figured that out. While we still have time, we should be encouraging (and also constructively criticizing) attempts to solve those two big problems. We should also continue to think about what happens if the kind of AI that we have now or in the very near future, should acquire superintelligence.
I agree with the author this much, that the values of our current AIs are in the right direction in various ways, and this improves the odds of a good outcome. But there are still various concerns, specific to AI takeover. What if an AI has deep values that are alien dispositions, and its humanism is simply an adaptation that will be shed once it no longer needs to get along with humans? What if there's something a bit wrong or a bit missing in the stew of values and dispositions instilled via training, system prompt, and conditioning? What if there's something a bit wrong or a bit missing in how it grounds its concepts, once it's really and truly thinking for itself?
We might also want to think about what happens to a human brain that takes over the world via AI infrastructure. If Elon makes himself emperor of known space via Neuralink and Grok, what are the odds that his transhuman form is good, bad, or just alien and weird, in what it wants?
I'm interested as to why?
I never heard of the book before, so I'm starting from a baseline of zero curiosity prior to this review. :-)
But also, I'm curious about how all the pieces fit together, and what else might be banging around in those almost 1000 pages. I'm wondering if "the signal" is a callback to the metavirus from Neal Stephenson's Snow Crash, and to William Burroughs's "language is a virus from outer space" (which Stephenson was undoubtedly referencing too). I don't quite trust your grounds for dismissing various concepts mentioned as impossible, and (as you say) concepts in SF can have other kinds of value even if they are literally impossible. Finally, whatever else it is, it is clearly a work of imagination, and such works can say something about their zeitgeist, or can be weird untimely works that stand alone.
Christoph Wetterich tried to make a shrinking universe work, though it seems to me it should result in the energy levels in atoms changing.
Your review actually makes me more curious about the book.
Could you give some examples of ontology pyramid schemes?
Also, can ontologies that are actually true, be the subject of pyramid schemes, as you define them?
If I take the view that AGI has existed since late 2022 (ChatGPT), I can interpret all this as a statement about the present: capital matters more than ever, during AGI.
After AGI is ASI, and at that point human capital is as valuable as ant capital or bear capital.
There's a 2009 interview with a transhumanist Australian academic where Egan hints at some of his problems with transhumanism (even while stating elsewhere that human nature is not forever, that he expects conscious AI in his lifetime, that "universal immortality" might be a nice thing, and so forth). Evidently some of it is pure intellectual disagreement, and some of it is about not liking the psychological attitudes or subcultural politics that he sees.
Do we know any Chinese AI forums or streamers, comparable to what's on Reddit and YouTube?
I have seen a poll asking "when will indefinite lifespans be possible?", and Eric Drexler answered "1967", because that was when cryonic suspension first became available.
Similarly, I think we've had AGI at least since 2022, because even then, ChatGPT was an intelligence, and it was general, and it was artificial.
(To deny that the AIs we have now have general intelligence, I think one would have to deny that most humans have general intelligence, too.)
So that's my main reason for very short timelines. We already crossed the crucial AGI threshold through the stupid serendipity of scaling up autocomplete, and now it's just a matter of refining the method, and attaching a few extra specialized modules.
What's the difference between "panology" and "science"?
By the start of April half the world was locked down, and Covid was the dominant factor in human affairs for the next two years or so. Do you think that issues pertaining to AI agents are going to be dominating human affairs so soon and so totally?
Hi - I would like you to explain, in rather more detail, how this entity works. It's "Claude", but presumably you have set it up in some way so that it has a persistent identity and self-knowledge beyond just being Claude?
If I understand correctly, you're trying to figure out what Xi would do with the unlimited power offered by an intent-aligned ASI, or how he would react to the prospect of such, etc.
Xi's character might matter, but I am impressed by the claim here that a competent Chinese ruler will be guided first by notions of good statecraft, with any details of personality or private life to be kept apart from their decisions and public actions.
I'm sure that Chinese political history also offers many examples of big personalities and passionate leaders, but that would be more relevant to times when the political order is radically in flux, or needs to be rebuilt from nothing. Xi came to power within a stable system.
So you might want to also ask how the Chinese system and ideology would respond to the idea of superintelligent AI - that is, if they are even capable of dealing with the concept! There must be considerable probability that the system would simply tune out such threatening ideas, in favor of tamer notions of AI - we already see this filtering at work in the West.
I suppose one possibility is that they would view AI, properly employed, as a way to realize the communist ideal for real. Communist countries always say that communism is a distant goal, for now we're building socialism, and even this socialism looks a lot like capitalism these days. And one may say that the powerbrokers in such societies have long since specialized in wielding power under conditions of one-party capitalism and mercantile competition, rather than the early ideal of revolutionary leveling for the whole world. Nonetheless, the old ideal is there, just as the religious ideals still exert a cultural influence in nominally secular societies descended from a religious civilization.
When I think about Chinese ASI, the other thing I think about, is their online fantasy novels, because that's the place in Chinese culture where they deal with scenarios like a race to achieve power over the universe. They may be about competition to acquire the magical legacy of a vanished race of immortals, rather than competition to devise the perfect problem-solving algorithm, but this is where you can find a Chinese literature that explores the politics and psychology of such a competition, all the way down to the interaction between the private and public lives of the protagonists.
Alexander Dugin speaks of "trumpo-futurism" and "dark accelerationism".
Dugin is a kind of Zizek of Russian multipolar geopolitical thought. He's always been good at quickly grasping new political situations and giving them his own philosophical sheen. In the past he has spoken apocalyptically of AI and transhumanism, considering them to be part of the threat to worldwide tradition coming from western liberalism. I can't see him engaging in wishful thinking like "humans and AIs coexist as equals" or "AIs migrate to outer space leaving the Earth for humans", so I will be interested to see what he says going forward. I greatly regret that his daughter (Daria Dugina) was assassinated, because she was taking a serious interest in the computer age's ontology of personhood, but from a Neoplatonist perspective; who knows what she might have come up with.
Started promisingly, but like everyone else, I don't believe in the ten-year gap from AGI to ASI. If anything, we got a kind of AGI in 2022 (with ChatGPT), and we'll get ASI by 2027, from something like your "cohort of Shannon instances".
For my part, I have been wondering this week, what a constructive reply to this would be.
I think your proposed imperatives and experiments are quite good. I hope that they are noticed and thought about. I don't think they are sufficient for correctly aligning a superintelligence, but they can be part of the process that gets us there.
That's probably the most important thing for me to say. Anything else is just a disagreement about the nature of the world as it is now, and isn't as important.
Perhaps he means something like what Keynes said here.
Your desire to do good and your specific proposals are valuable. But you seem to be a bit naive about power, human nature, and the difficulty of doing good even if you have power.
For example, you talk about freeing people under oppressive regimes. But every extant political system and major ideology, has some corresponding notion of the greater good, and what you are calling oppressive is supposed to protect that greater good, or to protect the system against encroaching rival systems with different values.
You mention China as oppressive and say Chinese citizens "can do [nothing] to cause meaningful improvement from my perspective". So what is it when Chinese bring sanitation or electricity to a village, or when someone in the big cities invents a new technology or launches a new service? That's Chinese people making life better for Chinese. Evidently your focus is on the one-party politics and the vulnerability of the individual to the all-seeing state. But even those have their rationales. The Leninist political system is meant to keep power in the hands of the representatives of the peasants and the workers. And the all-seeing state is just doing what you want your aligned superintelligence to do - using every means it has, to bring about the better world.
Similar defenses can be made of every western ideology, whether conservative or liberal, progressive or libertarian or reactionary. They all have a concept of the greater good, and they all sacrifice something for the sake of it. In every case, such an ideology may also empower individuals, or specific cliques and classes, to pursue their self-interest under the cover of the ideology. But all the world's big regimes have some kind of democratic morality, as well as a persistent power elite.
Regarding a focus on suffering - the easiest way to abolish suffering is to abolish life. All the difficulties arise when you want everyone to have life, and freedom too, but without suffering. Your principles aren't blind to this, e.g. number 3 ("spread empathy") might be considered a way to preserve freedom while reducing the possibility of cruelty. But consider number 4, "respect diversity". This can clash with your moral urgency. Give people freedom, and they may focus on their personal flourishing, rather than the suffering or oppressed somewhere else. Do you leave them to do their thing, so that the part of life's diversity which they embody can flourish, or do you lean on them to take part in some larger movement?
I note that @daijin has already provided a different set of values which are rivals to your own. Perhaps someone could write the story of a transhuman world in which all the old politics has been abolished, and instead there's a cold war between blocs that have embraced these two value systems!
The flip side of these complaints of mine, is that it's also not a foregone conclusion that if some group manages to create superintelligence and actually knows what they're doing - i.e. they can choose its values with confidence that those values will be maintained - that we'll just have perpetual oppression worse than death. As I have argued, every serious political ideology has some notion of the greater good, that is part of the ruling elite's culture. That elite may contain a mix of cynics, the morally exhausted and self-interested, the genuinely depraved, and those born to power, but it will also contain people who are fighting for an ideal, and new arrivals with bold ideas and a desire for change; and also those who genuinely see themselves as lovers of their country or their people or humanity, but who also have an enormously high opinion of themselves. The dream of the last kind of person is not some grim hellscape, it's a utopia of genuine happiness where they are also worshipped as transhumanity's greatest benefactor.
Another aspect of what I'm saying, is that you feel this pessimistic about the world, because you are alienated from all the factions who actually wield power. If you were part of one of those elite clubs that actually has a chance of winning the race to create superintelligence, you might have a more benign view of the prospect that they end up wielding supreme power.
I don't have a detailed explanation, but the user is posting a series of assignment or exam questions. Some of them are about "abuse". Gemini is providing an example of verbal abuse.
If I understand you correctly, you want to create an unprecedentedly efficient and coordinated network, made out of intelligent people with goodwill, that will solve humanity's problems in theory and in practice?
These are my thoughts in response. I don't claim to know that what I say here is the truth, but it's a paradigm that makes sense to me.
Strategic global cooperation to stop AI is effectively impossible, and hoping to do it by turning all the world powers into western-style democracies first is really impossible. Any successful diplomacy will have to work with the existing realities of power within and among countries, but even then, I only see tactical successes at best. Even stopping AI within the West looks very unlikely. Nationalization is conceivable, but I think it would have to partly come as an initiative from a cartel of leading companies; there is neither the will nor the understanding in the non-tech world of politics to simply impose nationalization of AI on big tech.
For these reasons, I think the only hope of arriving at a human-friendly future by design rather than by accident, is to solve the scientific, philosophical, and design issues involved, in the creation of benevolent superhuman AI. Your idea to focus on the creation of "digital people" has a lot in common with this; more precisely, I would say that many of the questions that would have to be answered, in order to know what you're doing when creating digital people, are also questions that have to be answered, in order to know how to create benevolent superhuman AI.
Still, in the end I expect that the pursuit of AI leads to superintelligence, and an adequately benevolent superintelligence would not necessarily be a person. It would, however, need to know what a person is, in a way that isn't tied to humanity or even to biology, because it would be governing a world in which that "unprecedented diversity of minds" can exist.
Eliezer has argued that it is unrealistic to think that all the scientific, philosophical, and design issues can be solved in time. He also argues that in the absence of a truly effective global pause or ban, the almost inevitable upshot is a superintelligence that reorganizes the world in a way that is unfriendly to human beings, because human values are complex, and so human-friendliness requires a highly specific formulation of what human values are, and of the AI architecture that will preserve and extrapolate them.
The argument that the design issues can't be resolved in time is strong. They involve a mix of perennial philosophical questions like the nature of the good, scientific questions like the nature of human cognition and consciousness, and avantgarde computer-science issues like the dynamics of superintelligent deep learning systems. One might reasonably expect it to take decades to resolve all these.
Perhaps the best reason for hope here, is the use of AI as a partner in solving the problems. Of course this is a common idea, e.g. "weak-to-strong generalization" would be a form of this. It is at least conceivable that the acceleration of discovery made possible by AI, could be used to solve all the issues pertaining to friendly superintelligence, in years or months, rather than requiring decades. But there is also a significant risk that some AI-empowered group will be getting things wrong, while thinking that they are getting it right. It is also likely that even if a way is found to walk the path to a successful outcome (however narrow that path may be), that all the way to the end, there will be rival factions who have different beliefs about what the correct path is.
As for the second proposition I have attributed to Eliezer - that if we don't know what we're doing when we cross the threshold to superintelligence, doom is almost inevitable - that's less clear to me. Perhaps there are a few rough principles which, if followed, greatly increase the odds in favor of a world that has a human-friendly niche somewhere in it.
Who said biological immortality (do you mean a complete cure for ageing?) requires nanobots?
We know individual cell lines can go on indefinitely, the challenge is to have an intelligent multicellular organism that can too.
It's the best plan I've seen in a while (not perfect, but has many good parts). The superalignment team at Anthropic should probably hire you.
Isn't this just someone rich, spending money to make it look like the market thinks Trump will win?
Doom aside, do you expect AI to be smarter than humans? If so, do you nonetheless expect humans to still control the world?