How much progress actually happens in theoretical physics?

As Adam Scherlis implies, the standard model turns out to be very effective at all the scales we can reach. There are a handful of phenomena that go beyond it - neutrino masses, "dark matter", "dark energy" - but they are weak effects that offer scanty clues as to what exactly is behind them. 

On the theoretical side, we actually have more models of possible new physics than ever before in history, the result of 50 years of work since the standard model came together. A lot of that is part of a synthesis that includes the string theory paradigm, but there are also very large numbers of theoretical ideas that are alternatives to string theory or independent of string theory. So if a decisive new phenomenon shows up, or if someone has a radical insight on how to interpret the scanty empirical clues we do have, we actually have more theories and models than ever before, that might be capable of explaining it. 

The idea that progress is stalled because everyone is hypnotized by string theory, I think is simply false, and I say that despite having studied alternative theories of physics, much much more than the typical person who knows some string theory. I think this complaint mostly comes from people who don't like string theory (Peter Woit) or who have an alternative theory they think has been neglected (Eric Weinstein). String theory did achieve a kind of hegemony within elite academia, but this was well-deserved, and meanwhile many competing research programs have had a foothold in academia too, to say nothing of the hundreds of physicists worldwide who have a personal theory that they write about, when they aren't doing other things like teaching. 

Most likely there are lost opportunities during that 50 years (like everyone else, I have my own ideas about neglected directions of research), but "do less string theory" is no guarantee that they would have been picked up. There are even those who would argue that there should have been more string theory of a certain kind (Lubos Motl used to say that field-theoretic phenomenologists should pay more attention to string theory, as a constraint and a guide in their model-building, and "stringking42069" says that the senior figures of string theory are holding the subject back by favoring work on their own little bandwagons, rather than bold and creative work on new directions and big unsolved problems). 

String theorist here. Progress has absolutely not stalled. The last big theoretical breakthrough was the AdS/CFT correspondence ( https://arxiv.org/abs/hep-th/9711200  ). This was our Transformers moment. This paper revolutioned the field, has now 20k+ citations, the most for any physics theory paper. It did completely solve the problem of quantum gravity for AdS spacetimes (negative curvature). There are too many things that have followed from this but I can give a few tidbits. For example it was understood that quantum black holes are the most chaotic objects in nature (https://arxiv.org/abs/1306.0622 https://arxiv.org/abs/1503.01409). A deep relationship between quantum gravity and quantum information theory was found (e.g, https://arxiv.org/abs/hep-th/0603001 https://arxiv.org/abs/1306.0533  https://arxiv.org/abs/1411.7041 ) which ultimately led recently to a solution of the black hole information paradox 50+ years after posed by Hawking! (https://arxiv.org/abs/2006.06872). There are caveats ofc, our universe is a dS spacetime (positive curvature) so current work is trying to extend AdS/CFT to cosmological spacetimes like our own.

The main reason string theory has bad press in the public is due to the fact that most top scientits don't have time to do public outreach (and their time is probably spent better elsewhere). So the only "physicists" who do public outreach are the ones who have nothing better to do... they are often bitter or disgrnuntled people whose subfield is becoming irrelevant and who have fallen behind. So they maintain blogs or publish books to feel more relevant saying things like "not even wrong" "physics is dead" "lost in math" etc. (there are exceptions ofc such as Brian Greene, Max Tegmark etc.).

The main thing I'd add is that string theory is not the problem. All the experimental low hanging fruit was picked decades ago. There are very general considerations that suggest that any theory of quantum gravity, stringy or otherwise, will only really be testable at the Planck scale. What this means in practice is that theoretical high-energy physics doesn't get to check its answers anymore.

I think there's still progress, and still hope for new empirical results (especially in astrophysics and cosmology), but it's much harder without a barrage of unexplained observations.

A lot of the effect is picking high-hanging fruit.

Like, go to phys rev D now. There's clearly a lot of hard work still going on. But that hard work seems to be getting less result, because they're doing things like carefully calculating the trailing-order terms of the muon's magnetic moment to get a change many decimal places down. (It turns out that this might be important for studying physics beyond the Standard Model. So this is good and useful work, definitely not being literally stalled.)

Another chunk of the effect is that you generally don't know what's important now. In hindsight you can look back and see all these important bits of progress woven into a sensible narrative. But research that's being done right now hasn't had time to earn its place in such a narrative. Especially if you're an outside observer who has to get the narrative of research third-hand.

First, you should distinguish theoretical physics from fundamental physics. For example, there is a theory of superconductivity, but it doesn't apply to high temperature superconductors, so there is room for more theories. The high status of fundamental physics is probably bad for theoretical physics, drawing people away from a huge range of open problems, but this isn't the fault of string theory. String theory compounded this by producing the illusion of opportunity, but it is hard to say how much is each problem.

Second, even if string theory is wrong, it can be useful. AdS is obviously wrong, but its ease of study can be useful. People go back and forth between general arguments and specific arguments. General arguments about what any theory of quantum gravity must look like are good. Checking that they can be made rigorous in specific theories like string theory is also good. Having more specific theories would be better.

Since physics separated from natural philosophy in the times of Newton, it has almost always^[1] progressed when new experimental data uncovered deficiencies in then-current understanding of the universe. During the Cold War unprecedentedly large amount of money were invested into experimental physics, and by the late 20th century all reasonably low hanging fruits have been picked (in the meantime the experiments have got absurdly expensive and difficult). I have also wrote on the topic at https://www.lesswrong.com/posts/CCnycGceT4HyDKDzK/a-history-of-the-future-2025-2040?commentId=KtusJZLAFDt4PW65R [LW(p) · GW(p)] and the thread below, check it out.

As of the string theory in particular, it represents just one significant school of thought very popular in the US but other theories share the same problem of lacking the experimental data to test against.

Also, the body of knowledge in physics has become so large that local progress made here and there is not really visible in the grand scheme of things anymore even if it's worth a Nobel Prize (while during the Second Industrial Revolution one discovery could, figuratively speaking, establish a new branch of science)

1. ^{^}

   Two notable exceptions that, IMHO, kind of support the rule are Maxwell's Equations and the General Relativity

Yes diminishing returns. For a purely theoretical field say pure mathematics this is even clearer. Take a fixed number of humans with a fixed intelligence (both average and outliers) then let mathematics advance. It will advance to the point that there is a vanishingly small number of people who can even understand the state of the art, with them then making little to no further progress. All else being equal the promising ideas are explored first etc.

More people just pushes you along the asymptote faster. If we took things even further, some knowledge would not even be live at one time. For example if there happened to be a few genius in a sub-field and it was in fashion, then humanities understanding of that sub field would advance, and be written down. However in 50 years or so no-one would actually understand the work, and it would require an unusual genius to be born to even take humanity knowledge to the edge of what is written down.

Given theoretical physics has such low useful bandwidth coming from experiments (Higgs is real, what else, countless null results upholding  the standard model don't help) then the same situation applies. We won't solve quantum gravity this side of the Singularity or massive human enhancement.

I have a second-handed source hearing this view from a theoretical physics 4th phd student at Stanford - he believes less breakthroughs nowadays as the field becomes more and more established, and this was exactly why he was a bit discouraged/sad. Not sure if things has changed, and that may or may not be his personal view.