What is the best paper explaining the superiority of Bayesianism over frequentism?

Well, there's a couple of issues here: first, logP(data|model) is a concave function for logistic regression, so unless logP(model) is also concave, the maximization may not reach the global optimum.

Secondly, the proper Bayesian thing to do would be to sample from the posterior, not maximize; for instance, in logistic regression the model is given by a vector of parameters denoted by theta. Suppose that we actually believed that the prior on theta was exp(-|theta|), where |theta| is the sum of the absolute values of the coordinates of theta. Then maximizing P(model|data) in this case will tend to give you solutions where most of the entries of theta are equal to 0, whereas the actual posterior places zero probability mass on such solutions.

Yeah, calling PCA frequentist may be a bit of a stretch (although it's certainly not Bayesian). I think ICA (independent components analysis) could legitimately be called frequentist though, as it solves the blind source separation problem under certain independence assumptions (I don't know that much about either of these though, so I could be wrong).

Do you accept that by Cox's theorems, probability theory is the normative theory of epistemology?

Not Cox's theorem, although the complete class theorem is more convincing (as well as dutch book arguments).

Do you accept that a "bayesian" method based on explicitly approximating ideal probability theory will always give a more accurate answer?

Only in the very weak sense that by the complete class theorem there exists a Bayesian method (or a limit of Bayesian methods) that does at least as well as whatever you're doing. So sure, if you really had infinite computational resources then you could find such a method and use it...but I think that has almost no bearing on practice. Certainly I think there are many situations where a prior is unavailable.

Do you accept that each of the examples above work because and to the extent that they (nonexplicitly) approximate the correct probability-theory answer (the bayes-structure argument)?

Almost certainly not, although maybe we should taboo "because". First of all, the "correct" probability-theory answer is not well-defined because the choice of both the prior and likelihood are both completely unconstrained. Secondly, I think the choice of whether to be Bayesian or frequentist is not nearly as important as e.g. the choice of likelihood function.

We can immediately see that building in the prior disallows aggregation of different information sources.

I don't think the prior is what allows aggregation of different information sources, you can do transfer learning with vanilla logistic regression if you choose the right set of features.

Only reporting the mode hides confidence interval and goes way off in the presence of skew.

I agree with this although "being Bayesian" is neither necessary nor sufficient to deal with this (but would probably help on average).

Bayesian logistic regression is easy and superior to max liklihood for most things.

What do you mean by "Bayesian logistic regression"?