R&D is a Huge Externality, So Why Do Markets Do So Much of it?

I think "R&D" is a misleading category - it comprises a LOT of activities with different uncertainty, type, scope, and timeframe of impact.  For tax and reporting purposes, a whole lot of not-very-research-ey software and other engineering is classified as "R&D", though it's more reasonably thought of as "implementation and construction".

Nordquist's "Innovation" measure is very different from economic reporting of R&D spending.  This makes the denominator very questionable in your thesis.

Perhaps more important, returns are NOT uniformly distributed.  Even successful "pure research" projects have a MIX of short/medium-term localized benefits and longer/broader impacts, and research insitutions are (mostly) pretty good at managing BOTH grants and licensing/product development as funding and "value capture" mechanisms.

William Nordhaus estimates that firms recover maybe 2% of the value they create by developing new technologies.

Isn’t this the wrong metric? 2% of the value of a new technology might be a lot of money, far in excess of the R&D cost required to create it.

Some observations from startup culture about under investment in r&d even in one of the world's supposed hot beds of innovation:

1: investors want a return soon and often inadvertantly push for lower variance strategies.

2: it is difficult to sell people on entirely new product categories.

3: people give poor feedback on new product categories and so it is up to the taste of the inventor to refine the product development.

4: legal structures are set up for benefit of those good at navigating legal structures, see why most politicians are lawyers. Many inventors wind up forced out of returns to their inventions.

5: manufacturing new product categories often requires modifications to existing manufacturing capacity, which hurts economies of scale. Often it is difficult to find anyone willing to help manufacturer your thing at all.

6: inventors are cognitively weird and thus socially weird, and fail many tacit social handshakes in the course of trying to coordinate with others.

7: innovators who are both weird and have the requisite taste have a hard time getting along with each other since they have strong opinions, including some strong wrong opinions.

8: there is competition between time spent innovating and time spent networking into groups that would support the business considerations.

9: selection of good business partners is a separate skill from innovating.

10: incumbents will use their existing and considerable cash flows to push back against new entrants, sometimes illegally.

11: consumers on average have short time horizons, so innovations that drive bigger improvements over time can lose out to the fast easy fix that kicks cans down the road.

12: if an innovation is both cheap and something a consumer only needs to buy once it can be hard to support the business.

13: established product areas can have a minefield of IP such that the obvious combinations of features would be legally prohibited from being sold without unavailable license deals.

14: an innovation that interacts with a politicized aspect of the economy, eg recycling, will face strong pushes from unexpected directions unrelated to what the product tries to physically do.

15: an innovation that mainly helps people with problems that involve social desirability bias will be hard to market.

16: innovations that put people out of a job have a built in counter lobbying force.

They model the returns to R&D spending like this: imagine stopping all R&D spending for a single year. You’d save several hundred billion dollars upfront, but there would be no economic growth,¹ so we’d miss out on a ~2% increase in per capita GDP.

It's a cute argument and I like it, though I note that probably-most economic innovation does not come from things-called-R&D-spending. For instance, I'd expect selection pressure on businesses is a major mechanism. People try things and see what makes money, and that's not usually classified as "R&D".

When the government wants electric car production it has a choice to invest directly into electric car R&D or to subventionize electric cars. The United States spends much more money on subventions than on R&D.

I would guess that if you ask the lawmakers they would say something like "Electric car companies are much better at investing money into developing better cars than the government happens to be, so the subventions are the more effective policy tool than paying the same amount of money for basic research".

I don't think this is a pure matter of incentives but simply of the ideology of privatization. It might be wrong and in reality direct government R&D is more effective than subventions but that's an argument you have to make more directly.

The owner of a firm investing in R&D doesn’t account for all the benefits their technology might bring to non-paying consumers and firms, but they do care about the benefits that R&D will bring to the firm long into the future, even after their death. One part of this is that owners don’t face term limits that incentivize pump-and-dump attempts to garner voter support.

This does not match my expectations, even if it agrees with how I would feel were I the owner.

For example, the top ten Nasdaq companies spent ~$222B between them on R&D, which is almost half of the $463B spend from the private sector.

But I notice these are publicly traded companies, where the owner is in fact the shareholders. The average holding period for stocks is less than a year; if memory serves in 2020 it was less than 6 months. The average shareholder definitely does not care about the value of R&D to the firm long after their deaths, or I suspect any time at all after they sell the stock.

I notice that Amazon and Tesla are both on the list, and maybe by owner we really mean founder; I could easily see Bezos and Musk feeling that way about R&D at their companies. But most of R&D spend from the private sector is not from founder-run companies; it is from professionally managed companies with hired CEOs and executives.

On that note, I feel like I have seen plenty of cases where executives and CEOs do precisely the pump-and-dump phenomenon on the project side. Maybe executives have longer tenures than legislators, so it makes more sense for them to take a longer view? Yet when I check, CEOs have on average shorter tenure than Congress does: the CEO has 7.2 years; Representatives and Senators have 8.9 and 11 years respectively. I also note that the CEO has much more power over the budget than legislators do; the legislator at least requires the agreement or indifference of a majority of the legislature to get R&D stuff added, whereas the CEO can very often decide budget priorities unilaterally.

That being said, this doesn't seem to change the overall thrust of the post. My suspicion lands on the 2% captured value number as being misleading; for the individual company, 2% of a huge number can easily be more than 98% of a much smaller number. I'm also curious about how that number 2% is built, so my next step here is to check out the Nordhaus paper more deeply.

I wonder if the point about relative investment also holds if one restricts one's perspective to technologies needed to ensure security, i.e. military gear and similar technologies.

1. Categories

Fundamental Research = State
Applied Research = Companies

.. is a common paradigm, and - while grossly too simplified - makes some sense: the latter category has  more tangible outputs, shorter payback etc. In line with @Dagon [LW · GW]'s comment, at the very least these two broad categories would have to be split for a serious discussion of whether 'too much' or 'too little' is done by gvmt and/or companies.

2. Incentives!

I've worked in a research startup and saw the same dollar go much further in producing high-quality research outputs than what I've directly experienced in some places (and from many more places observed) in the exact same domain in state-sponsored research (academia) where there is often a type Resource Curse dynamics.
My impression is, these observations generalize rather well (I'm talking about a general tendency; needless to say, there are many counterexamples; often those wanting to do serious research are exactly attracted by public research opportunities, where they can do great work). Your explanation leaves out this factor; it might explain a significant part of the reluctance of the state to spend more on R&D.

This does not mean the state should not do more (or support more) R&D, but I think there are very important complexities the post leaves out, limiting the explanation power.

One reason that economically rational firms might produce "too much" R&D for their shareholders: competition! In many industries, if you stop innovating your company will fall behind quickly, and your customers will go elsewhere.

Happily this Red Queen's Race has large positive externalities, and so coordinating to reduce such investments is generally illegal ('restraint of trade').