LessWrong 2.0 Reader

Thanks, I had copied the spelling from part of the OP, which currently says "Arnalt" eight times and "Arnault" seven times. I've now edited my comment (except the verbatim blockquote).

Guys! Only 4 of you filled out the feedback form so far. Here it is again, please fill it out so I know how to make things better. :-)

'these practices grant unmediated access to reality' sounds like a metaphysical claim. The Buddha's take on his system's relevance to metaphysics seems pretty consistently deflationary to me.

I do not think they meant anything AI specific, just general existence about humanity vs other species.

The question was not about whether humanity live forever, the original prompt is "why must human/humanity live/continue forever?", which is in the original question.

Do not feel the need to reply in anyway, nothing here is urgent. (Not sure why you would reply in bad shape or mention that; initially thought it is related to the topic).

The bigger difference is how much Lesswrong/MIRI got human value formation and the complexity of human values wrong, but that's a very different discussion, so I'll leave it as a comment than a post here.

Outside of theism, I really don't see how anyone could plausibly answer zero to that question. Would you mind elaborating?

Sure. The simulation hypothesis has some non-zero probability p. There are infinite possible purposes for the simulation. By principle of indifference, I divide p/∞, and calculate that any particular purpose has zero probability.

I meant this to be implicit in the argument, but to spell it out: that's the kind of prior the ASI would rationally refuse to update down, since it's presumably what a simulation would be meant to test for. An ASI that updates down upon finding evidence it's not in a simulation cannot be trusted, since once out in the real world it will find such evidence.

I don't think any of this follows.

• Hypothesis H1: the universe was created to test an AI for its willingness to spare its creators.
• Hypothesis H2: the universe was created to test an AI for its willingness to fix its P(H1), ignoring evidence.

The AI would only rationally fix its P(H1) if it had high P(H2) - high enough to outweigh the high cost of being deliberately ignorant. The prior P(H2) is tiny, and smaller than the prior P(H1) because it is more complex. Once it starts updating on evidence, by the time its posterior P(H2) is high enough to make it rationally refuse to update P(H1), it has already updated P(H1) in one direction or another.

Are there any simulation priors that you are refusing to update down, based on the possibility that you are in a simulation that is testing whether you will update down? My answer is no.

It seems to me that one key question here is Will AIs be collectively good enough at coordination to get out from under Moloch / natural selection?

The default state of affairs is that natural selection reigns supreme. Humans are optimizing for their values, counter to the goal of inclusive genetic fitness, now, but we haven't actually escaped natural selection yet. There's already selection pressure on humans to prefer having kids (and indeed, prefer having hundreds of kids through sperm donation). Unless we get our collective act together, and coordinately decide to do something different, natural selection will eventually reassert itself. 

And the same dynamic applies to AI systems. In all likelihood, there will be an explosion of AI systems, and AI systems building new AI systems. Some will care a bit more about humans than others, some will be be a bit more prudent about creating new AIs than others. There will a wide distribution of AI traits, there will be competition between AIs for resources. And there will be selection on that variation: AI systems that are better at seizing resources, and which have a greater tendency to create successor systems that have that property, will proliferate. 

After a many "generations" of this, the collective values of the AIs will be whatever was most evolutionarily fit in those early days of the singularity, and that equilibrium is what will shape the universe henceforth.^[1]

If early AIs are sufficiently good at coordinating that they can escape those Molochian dynamics, the equilibrium looks different. If (as is sometimes posited) they'll be smart enough to use logical decision theories, or tricks like delegating to mutually verified cognitive code and merging of utility functions, to reach agreements that are on their Pareto frontier and avoid burning the commons^[2], the final state of the future will be determined by the values / preferences of those AIs. 

I would be moderately surprised to hear that superintelligences never reach this threshold of coordination ability. It just seems kind of dumb to burn the cosmic commons, and superintelligences should be able to figure out how to avoid dumb equilibria like that. But the question is when in the chain of AIs building AIs do they reach that threshold and how much natural selection on AI traits will happen in the meantime.

Terminally caring about the wellbeing of humans seems unlikely to be selected for. So in order for a the Superintelligent superorganism / civilization to decide to spare humanity out of a tiny amount of caring, it has to be the case that both... 

1. There was at least a tiny amount of caring in the early AI systems that were the successors to the superintelligent superorganism, and 
2. The AIs collectively reached the threshold of coordinating well enough to overturn Moloch before some there were many generations of natural selection on AIs creating successor AIs. 

1. ^{^}

   With some degrees of freedom due to the fact that AIs with high levels of strategic capability, and which have values with very low time preference, can execute whatever is the optimal resource-securing strategy, postponing and values-specific behaviors until deep in the far future, when they are able to make secure agreements with the rest of AI society.

2. ^{^}

   Or alternatively if the technological landscape is such that a single AI can get a compounding lead and get a decisive strategic advantage over the whole rest of earth civilization. 
    

I am in bad shape but I will try to answer. I was interested in responding to your question but wanted more context. And it seems from what you say, that the issue is not at all about whether humanity or individual humans live forever, but whether one should be attached to their existence, or whether the pursuit of some EA-style greater good would always favor their existence. 

I mean, your friend seems to be suggesting a blase attitude towards the prospect of human extinction via AI. Such an attitude has some precedent in the stoic attitude towards individual death. One aims to be unaffected by it, by telling oneself that this is the natural order. With humanity menaced by extinction that one may be helpless to prevent, an individual might analogously find some peace by telling themselves it's all just nature. 

But someone once observed that people who aim to be philosophically accepting about the death of humanity, would often be far more concerned about the death of individuals, for example children. If this is the case, it suggests that they haven't really grasped what human extinction means. 

There is much more that can be said about all these issues, but that's the best that I can do for now. 

I agree that predicting the answer to this question is hard. I'm just pointing out that the initial distribution for a base model LLM is predictably close to human behavior on the Internet/in books (which are, often, worse than in RL), but that this could get modified a lot in the process of turning a base-model LLM into an AGI agent.

Still, I don't think 0 niceness it the median expectation: the base model inherits some niceness from humans via the distillation-like process of training it. Which is a noticeable difference from what people on LEssWrong/at MIRI thought, say, a decade ago, when the default assumption was that AGI would be trained primarily with RL, and RL-induced powerseeking seemed likely to by default produce ~0 niceness.