It's hard to use utility maximization to justify creating new sentient beings

People want different things, and the different possible disagreement resolving mechanisms include the different varieties of utilitarianism.

In this view, the fundamental issue is whether you want the new entity to be directly counted in the disagreement resolving mechanism. If the new entity is ignored (except for impact on the utility functions of pre-existing entities, including moral viewpoints if preference utility is used*), then there's no need to be concerned with average v. total utilitarianism.

A general policy of always including the new entity in the disagreement resolving mechanism would be extremely dangerous (utility monsters). Maybe it can be considered safe to include them under limited circumstances, but the Repugnant Conclusion indicates to me that the entities being similar to existing entities is NOT sufficient to make it safe to always include them.

(*) hedonic utility is extremely questionable imo - if you were the only entity in the universe and immortal, would it be evil not to wirehead?

I don't know why the trade-off between population-size and average utility feels like it needs to be a mathematically justified, that function seems to be as much determined by arbitrary evolutionary selection as the rest of our utility functions.

I just thought of an argument that pulls toward average utilitarianism. Imagine I'm about to read a newspaper which will tell me the average happiness of people on Earth: is it 8000 or 9000 "chocolate equivalent units" per person? I'd much rather read the number 9000 rather than 8000. In contrast, if the newspaper is about to tell me whether the Earth's population is 8 or 9 billion people, I don't feel any strong hopes either way.

Of course there's selfish value in living in a more populous world, more people = more ideas. But I suspect the difficulty of finding good ideas rises exponentially with their usefulness, so the benefit you derive from larger population could be merely logarithmic.

I personally think that something more akin to minimum utilitarianism is more inline with my intuitions. That is, to a first order approximation, define utility as (soft)min(U(a),U(b),U(c),U(d)...) where a,b,c,d... are the sentients in the universe. This utility function mostly captures my intuitions as long as we have reasonable control over everyone's outcomes, utilities are comparable, and the number of people involved isn't too crazy.

I think I have a pretty simple solution to this: treat 0 as the point when each being is neither happy nor unhappy. Negative numbers are fine. You can still take the sum. In the example, this seems like just subtracting 10 from everyone, which is 0 in the dogless state and -3 with the dog. Thus: no puppy.

The puppy example seems pretty simple.  Nonexistent things don't have preferences, so that cell in the table is "n/a".  This is the same result as if you just ignored the heisenpuppy's value, but it wouldn't have to be (for instance, 14+2+5 > 10+10, so even if cedric only liked dogs a tiny amount, it'd be a net benefit to bring the dog into existence, but wouldn't be if the dog's happiness were unconsidered).

The rat king is similar to https://plato.stanford.edu/entries/repugnant-conclusion/ , but would be much stronger if you show the marginal decision, without the implication that if rats have a litter, that decision will continue to apply to all rats, regardless of situation.  Say, there's 100 rats that are marginally happy because there's just enough food.  Should they add 1 to their population?

I think the simulator is too far removed from decisions we actually make.  It's not a very good intuition pump because we don't have instinctive answers to question.  Alternately, it's just an empirical question - try both and see which one is better.

Your realistic examples are not extrapolate-able from the simple examples.  Chickens hinges on weighting, not on existential ethics.  There are very few people who claim it's correct because it's best for the chickens (though some  will argue that it is better for the chickens, that's not their motivation).   There are lots who argue that human pleasure is far more important than chicken suffering.  

The parenthood question is murky because of massive externalities, and a fallacy in your premises - the impact on others (even excluding the potential child) is greater than on the parent.  Also, nobody's all that utilitarian - in real decisions, people prioritize themselves.