Is backwards causation necessarily absurd?

causation might be in the map rather than the territory

Of course it is. There is no atom of causation anywhere. It's a tool for embedded agents to construct useful models in an internally partially predictable universe.

"Backward causation" may or may not be a useful model at times, but it is certainly nothing but a model.

As a trained (though not practicing) physicist, I can see that you are making a large category error here. Relativity neither adds to not subtracts from the causation models. In a deterministic Newtonian universe you can imagine backward causation as a useful tool. Sadly, its usefulness it rather limited. For example, the diffusion/heat equation is not well posed when run backwards, it blows up after a finite integration time. An intuitive way to see that is that you cannot reconstruct the shape of a glass of water from the puddle you see on the ground some time after it was spilled. But in cases where the relevant PDEs are well posed in both time directions, backward causality is equivalent to forward causality, if not computationally, then at least in principle.

All that special relativity gives you is that the absolute temporal order of events is only defined when they are within a lightcone, not outside of it. General relativity gives you both less and more. On the one hand, the Hilbert action is formulated without referring to time evolution at all and poses no restriction on the type of matter sources, be they positive or negative density, subluminal or superluminal, finite or singlular. On the other hand, to calculate most interesting things, one needs to solve the initial value problem, and that one poses various restrictions on what topologies and matter sources one can start with. On the third hand, there is a lot of freedom to define what constitutes "now", as many different spacetime foliations are on equal footing.

If you add quantum mechanics to the mix, the Born rule, needed to calculate anything useful regardless of one's favorite interpretation, breaks linearity and unitarity at the moment of interaction (loosely speaking) and is not time-reversal invariant.

The entropic argument is also without merit: there is no reason to believe that entropy would decrease in a "high-entropy world", whatever that might mean. We do not even know how observer-independent entropy is (Jaynes argued that apparent entropy depends on the observer's knowledge of the world).

Basically, you are confusing map and territory. If backward causality helps you make more accurate maps, go wild, just don't claim that you are doing anything other than constructing models.

I feel that questions like this have a hard time escaping confusion because the notion of linear time is so deeply associated with causality already.

Could you point me to the arguments about a high-entropy universe being expected to decrease in entropy?

I don't think backwards causation is absurd, more or less for the reasons you sketch. Another minor reason: Some philosophers like "effective strategy" accounts of causation, according to which we define causation via its usefulness for agents trying to achieve goals. On these accounts, backwards causation is pretty trivial--just suppose you live in a deterministic universe and your goal is to "make the state of the universe at the Big Bang such that I eat breakfast tomorrow." The philosopher Gary Drescher argues something similar in Good and Real if I recall correctly.

That said, I don't think we are really explaining or de-confusing anything if we appeal to backwards causation to understand Newcomb's Problem or argue for a particular solution to it.

Another relativistic argument against time flowing is that simultaneity is only defined relative to a reference frame. Therefore, there is no unified present which is supposed to be what is flowing.

Relativity does not make the arrow of time relative to observer. Events in one's future light cone remain in their future light cone also from a perspective of someone else.