Why Study Physics?

I think the title should be "why study physicists" not "why study physics". Because what you are describing is a gift certain physicists have, and others do not. I had it in high school (often when the teacher would state a problem in class, the only thing that was obvious from the beginning was the answer, not how to get to it), and it saved my bacon in grad school a few times many many years later. Recently it took my friend and me about 5 min of idle chatting to estimate the max feasible velocity of a centrifugal launch system, and where the bottlenecks might be (surprisingly, it is actually not the air resistance, and not the overheating, but the centrifugal g forces before launch which make single-stage-to-orbit impossible). John Wheeler famously said something like "only do a calculation once you know the result." Einstein knew what he wanted out of General Relativity almost from the start, and it took him years and years to make the math work. This pattern applies in general, as well. A physicist has a vague qualitative model that "feels right", and then finds the mathematical tools to make it work quantitatively, whether or not those tools are applied with any rigor. I don't know if this skill can be analyzed or taught, it seems more like artistic talent.

The Scientific Method played a role—physicists could check their results against real-world data—but that’s mostly just checking the answer. The hard part was to figure out which answer to check in the first place, and that involved informal mathematical arguments.

Do you expect these less legible techniques to generalize well outside of physics, especially places where checking the answer is impossible or prohibitively expensive? Maybe the reason these techniques are widely and successfully used in physics is because the answers can eventually be checked, which limit the extent to which the techniques can be misused/abused?

I think the use of symmetries for causal inference is an important insight in physics. Without constraints, there are an enormous amount of possible causal theories that can explain any given set of observations. But adding symmetries can often pin down the theory uniquely, and in a testable way (because the symmetry would fail for almost all the ways the theory could be false).

It also turns out that there's an obscure method in the causal inference literature that I think can be analogized to this. If you've got two variables $A$ and $B$, then the causal relationship $A\to B$ or $B\to A$ can be hard to figure out, just from their distribution $P(A,B)$. However, if you can embed them into a family $A_i$ and $B_i$ which have sufficiently different distributions, but which all share the mechanisms, then likely precisely one of $P\left(A_i|B_i\right)$ and $P\left(B_i|A_i\right)$ will be constant as a function of $i$, with the constant one being the one that encodes the direction of causality. I think this is a special-case of the physics method of requiring theories to respect symmetries, where in this case the symmetry group is the group of permutations on $i$.

I'm wondering about the different types of intuitions in physics and mathematics.

What I remember from prepa (two years after high school where we did the full undergraduate program of maths and physics) was that some people had maths intuition (like me) and some had physics intuition (not me). That's how I recall it, but thinking back on it, there were different types of maths intuitions, which correlated very differently with physics intuition. I had algebra intuition, which means I could often see the way to go about algebraic problems, whereas I didn't have analysis intuition, which was about variations and measures and dynamics. And analysis intuition correlated strongly with physical intuition.

It's also interesting that all your examples of physicist using informal mathematical reasoning successfully ended up being formalized through analysis.

This observation makes me wonder if there are different forms of "informal mathematical reasoning" underlying these intuitions, and how relevant each one is to alignment.

• An algebra/discrete maths intuition which is about how to combine parts into bigger stuff and reversely how to split stuff into parts, as well as the underlying structure and stuffs like generators. (Note that "the deep theory of addition" discussed recently is probably there)
• An analysis/physics intuition which is about movement and how a system reacts to different changes.

Also the distinction becomes fuzzy because there's a lot of tricks which allow one to use a type of intuition to study the objects of the other type (things like analytic methods and inequalities in discrete maths, let's say, or algebraic geometry). Although maybe this is just evidence that people tend to have one sort of intuition, and want to find way of applying it at everything.

Physics also seems to help with clear philosophical thinking, and has lots of unintuitive stuff that trains the skill of looking past your models and into the Real Thing. Of Deep Fundamental Principles, I also think physics has some of the easier ones to see, like the conservation principles (once you get why a physicist can be so confident of the non-existence of a perpetual motion machine, you can start to imagine how there could be other Deep Principles which could justify seemingly excessive confidence about other complicated domains).

On the other hand, mediocre physicists tend to be too arrogant, and the current generation of physicists seems to have lost the way in some important (but hard to pin down) sense.

Related work:

• Armstrong on informal probabilistic proofs of the Goldbach hypothesis and Fermat's last theorem [LW · GW] 
• Arguments against P = NP in Arbital (disclaimer: I wrote these)
• Gwern discussing mathematical error, specifically about the puzzle of why most math theorems are correct even if errors in proofs are relatively common 

Also, while this post focuses a lot on Physics, my experience is that top level math people are quite comfortable with informal math reasoning.

we don’t have a step-by-step checklist to follow in order to use informal mathematical arguments

If we did, the checklist would define a form and the mathematical arguments would become formal.

Terrence Tao uses the term post-rigorous to describe the sort of argument you're talking about.  It's one of three stages.  In the pre-rigorous stage, concepts are fuzzy and expressed inexactly.  In the rigorous stage, concepts are precisely defined in a formal manner.  In the post-rigorous stage, concepts are expressed in a fuzzy and inexact way for the sake of efficiency by people who understand them on a rigorous level; key details can be expressed as rigorously as necessary but the irrelevant details of a full proof are omitted.

This feels correct; my experience of physics+calculus in high school was that it was harder than any other class I took but made me [feel on the inside] a lot smarter, like the insights were important and unlocked new patterns of thought, even though everyone else in the class had a much easier time with the material (high-level elective with calculus as prerequisite screened out all but the most committed).

I have fond memories. I would read three paragraphs in the text book, ask my physics-gifted friend, "wtf?" and ze would say, "Oh, that just means x," and I would say, "Oh, it seems obvious now--why didn't they just say x??"

My friend was VERY patient with me.

One major pattern of thought I picked up from (undergraduate) physics is respect for approximation. I worry that those who have this take it for granted, but the idea of a rigorous approximation that's provably accurate in certain limits, as opposed to a casual guess, isn't obvious until you've encountered it.

Yet there’s clearly something making informal mathematical arguments correlate quite highly with truth (if imperfectly), because they work well in practice!

Yes. It's empiricism. If you use informal arguments in an apriori way, you get philosophy.

Physics also seems to help with clear philosophical thinking, and has lots of unintuitive stuff that trains the skill of looking past your models and into the Real Thing

Well, it's got lots of unintuitive stuff. Much more so than armchair metaphysics ever came up with.

Having a broad selection of theories choose from is, in a way, helpful to the process of honing in on the one true theory ... But it puts a lot more stress on the honing-in part of the process.