With enough wealth to trivially save all monkeys, no monkeys would go extinct, provided we care even a little bit more than precisely not at all.

I'm confused about this point. Perhaps you mean wealth in a broad sense that includes "we don't need to worry about getting more wood." But as long as wood is a useful resource that humans could use more of to acquire more wealth and do other things that we value more than saving monkeys, then we will continue to take wood from the monkeys. Likewise, even if an AGI values human welfare somewhat, it will still take our resources as long as it values other things more than human welfare.

I found the monkey example much more compelling than "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." Taking resources from humans seems more likely than using humans as resources.

something trained on human culture might retain at least a tiny bit of compassion on reflection

This depends on where "compassion" comes from. It's not clear that training on data from human culture gets you much in the way of human-like internals. (Compare: contemporary language models know how to say a lot about "happiness", but it seems very dubious that they feel happiness themselves.)

These are good points. Maybe we'll align these things enough to where they'll give us a little hamster tank to run around in.

I agree that I did not justify this claim and it is controversial in the ML community. I'll try to explain why I think this.

First of all, when I say an AI 'wants things' or is 'an agent' I just mean it robustly and autonomously brings about specific outcomes. I think all of the arguments made above apply with this definition and don't rely on anthropomorphism.

Why are we likely to build superintelligent agents?
1. It will be possible to build superintelligent agents
I'm just going to take it as a given here for the sake of time. Note: I'm not claiming it will be easy, but I don't think anyone can really say with confidence that it will be impossible to build such agents within the century.
2. There will be strong incentives to build superintelligent agents
I'm generally pretty skeptical of claims that if you just train an AI system for long enough it becomes a scary consequentialist. Instead, I think it is likely someone will build a system like this because humans want things and building a superintelligent agent that wants to do what you want pretty much solves all your problems. For example, they might want to remain technologically relevant/be able to defend themselves from other countries/extraterrestrial civilizations, expand human civilization, etc. Building a thing that 'robustly and autonomously brings about specific outcomes' would be helpful for all this stuff.

In order to use current systems ('tool AIs' as some people call them) to actually get things done in the world, a human needs to be in the loop, which is pretty uncompetitive with fully autonomous agents.

I wouldn't be super surprised if humans didn't build superintelligent agents for a while -- even if they could. Like I mentioned in the post, most people prob want to stay in control. But I'd put >50% credence on it happening pretty soon after it is possible because of coordination difficulty and the unilateralist curse.

Marginal utility is a thing.

This alone isn't enough, and in the past I didn't believe the conclusion. The additional argument that leads to the conclusion is path-dependence of preferred outcomes. The fact that human civilization currently already exists is a strong argument for it being valuable to let it continue existing in some form, well above the motivation to bring it into existence if it didn't already exist. Bringing it into existence might fail to make the cut, as there are many other things that a strongly optimized outcome could contain, if its choice wasn't influenced by the past.

relatively hot planet in a high-oxygen atmosphere with lots of water

But atoms? More seriously, the greatest cost is probably starting expansion a tiny bit later, not making the most effective use of what's immediately at hand.