Does that make sense?

In a word, no.

I believe you are thinking of infinity as a number, and that's always a mistake. I think that what you're trying to say with your left-hand graph is that, given infinite utility, probability is a tiebreaker, but all infinite-utility options dominate all finite utilities. But this treats "infinity" as a binary quality which an option either has or not.

Consider two different Pascal's muggers: One offers you a 1% probability of utility increasing linearly in time, the other, a 1% chance of utility increasing exponentially with time. Clearly both options "are infinite"; equally clearly, you prefer the second one even though the probabilities are the same. They occupy the same point on your left-hand graph. But by your suggested decision procedure you would choose the linearly-increasing option if the first mugger offered even an epsilon increase in probability; and this is obviously Weird. It gives you a smaller expected utility at almost all points in time!