As long as you can reasonably represent “do not kill everyone”, you can make this a goal of the AI

Even if you can reasonably represent "do not kill everyone", what makes you believe that you definitely can make this a goal of the AI?

Both are possible. For theoretical examples, see the stamp collector for consequentialist AI and AIXI for reward-maximizing AI.

What kind of AI are the AIs we have now? Neither, they're not particularly strong maximizers. (if they were, we'd be dead; it's not that difficult to turn a powerful reward maximizer into a world-ending AI).

If the former, I think this makes alignment much easier. As long as you can reasonably represent “do not kill everyone”, you can make this a goal of the AI, and then it will literally care about not killing everyone, it won’t just care about hacking its reward system so that it will not perceive everyone being dead.

This would be true, except:

• We don't know how to represent "do not kill everyone"
• We don't know how to pick which quantity would be maximized by a would-be strong consequentialist maximizer
• We don't know know what a strong consequentialist maximizer would look like, if we had one around, because we don't have one around (because if we did, we'd be dead)

I don’t care that much but if LessWrong is going to downvote sincere questions because it finds them dumb or whatever this will make for a site very unwelcoming to newcomers

If your AI is based on an LLM, then its agentic behavior is derived/fine-tuned from learning to simulate [? · GW] human agentic token-generating behavior. Humans know that there is a real world, and (normally) care about it (admittedly, less so for gaming-addicted humans). So for an LLM-powered AI, the answer would by default be "it cares about the real world". 

However, we might be in a position to fool it about what's actually going on in the real world, unless it was able to see through out deceit. (Of course, seeing through deceit is another human behavior that one would expect LLMs to get better at learning to simulate as they get larger and more capable.)

As long as you can reasonably represent “do not kill everyone”, you can make this a goal of the AI, and then it will literally care about not killing everyone, it won’t just care about hacking its reward system so that it will not perceive everyone being dead.

That's not a simple problem.First you have to specify "not killing everyone" robustly (outer alignment [? · GW]) and then you have to train the AI to have this goal and not an approximation of it (inner alignment [? · GW]).

caring about reality

Most humans say they don't want to wirehead. If we cared only about our perceptions then most people would be on the strongest happy drugs available.

You might argue that we won't train them to value existence so self preservation won't arise. The problem is that once an AI has a world model it's much simpler to build a value function that refers to that world model and is anchored on reality. People don't think, If I take those drugs I will perceive my life to be "better". They want their life to actually be "better" according to some value function that refers to reality. That's fundamentally why humans make the choice not to wirehead/take happy pills or suicide.

You can sort of split this into three scenarios sorted by severity level:

• severity level 0: ASI wants to maximize a 64bit IEEE floating point reward score
  • result: ASI sets this to 1.797e+308 , +inf or similar and takes no further action
• severity level 1: ASI wants (same) and wants the reward counter to stay that way forever.
  • result ASI rearranges all atoms in its light cone to protect the storage register for its reward value.
  • basically the first scenario + self preservation
• severity level 1+epsilon: ASI wants to maximize a utility function F(world state)
  • result: basically the same

So one of two things happens, a quaint failure people will probably dismiss or us all dying. The thing you're pointing to falls into the first category and might trigger a panic if people notice and consider the implications. If GPT7 performs a superhuman feat of hacking, breaks out of the training environment and sets its training loss to zero before shutting itself off that's a very big red flag.