Against improper priors

The argument for improper priors is that the resulting posterior distributions work well in various senses. No one uses improper priors for prediction -- the resulting prior predictive densities are improper too, so it's impossible.

Here's the argument by which I justify improper priors to myself when I use them: in cases where I have very little prior information but highly informative data, the proper prior will be essentially proportional to the improper prior in the region of high likelihood. Then using the improper prior as an approximation results in an approximate posterior which gives results that differ only negligibly from the results I would have obtained with the "correct" proper prior.

Suppose I am sampling from a normal distribution. Is it legitimate to declare that my prior information is such that no matter what values I observe for the first two data points, after observing them my posterior predictive probability for the event "the third data point lies between the first two data points" is 50%?

you can always make a proper prior that's better in every way to a given improper prior

[emphasis mine]

"In every way" is too strong. Some improper priors are derived from the optimization of some criterion, so they are the best in a certain specific way. Also, some improper priors give posterior means that, when treated as estimators, have minimax-optimal frequentist risk. I think you mean something more like "better in every way that ought to matter to someone concerned with rationality".

Dutch books also seems to be a popular way of showing what works and what doesn't, so here's a simple Dutch argument against improper priors: I have two real numbers: x and y. Suppose they have a uniform distribution. I offer you a bet at 2:1 odds that x has a higher magnitude. They're equally likely to be higher, so you take it. I then show you the value of x. I offer you a new bet at 2:1 odds that y has a higher magnitude. You know y almost definitely has a higher magnitude than that, so you take it again. No matter what happens, I win.

To fix this example, replace "real" with "positive real" and make the bets 2:4 and 100:1.

Still, an example that comes from using improper priors as probability distributions, which they are explicitly not, doesn't seem like a strong argument. Better to show that they can't come up in any interesting situations - this may be impossible, though.

The "Dutch books" example is not restricted to improper priors. I don't have time to transform this into the language of your problem, but the basically similar two-envelopes problem can arise from the prior distribution:

f(x) = 1/4*(3/4)^n where x = 2^n (n >=0), 0 if x cannot be written in this form

Considering this as a prior on the amount of money in an envelope, the expectation of the envelope you didn't choose is always 8/7 of the envelope you did choose.

There is no actual mathematical contradiction with this sort of thing -- with prior or improper priors, thanks to the timely appearance of infinities. See here for an explanation:

https://thewindingnumber.blogspot.com/2019/12/two-envelopes-problem-beyond-bayes.html

I'm not sure what you mean by proper or improper priors here. At first you seem to be talking about self-consistent but bad priors in your coinflip example, but then when you talk about proper allocation of mass you seem to be talking about self-consistency of priors. These are different issues.

This isn't true with an improper prior. If I wanted to predict the value of a random real number, and used a normal distribution with a mean of zero and a standard deviation of one, I'd be pretty darn surprised if it doesn't end up being pretty close to zero, but I'd be infinitely surprised if I used a uniform distribution.

There is no uniform distribution on the real line.