We need a career path for invention

I definitely agree that the idea of unconstrained "invention" is not well supported in society, but the hypothesis makes me go "huh?"

Science discovers new knowledge; invention creates useful machines, chemicals, processes, or other products; and business produces and distributes these products in a scalable, self-sustaining way.

Is the place you use the word "invention" not engineering? For most types of engineering, undergrad students are taught science for two years (it is new knowledge to them), they're taught how to usefully apply that knowledge for a year and a half, and then they have a final semester or two explaining how that knowledge can be used to achieve some business goals. 

In other words, "career that applies scientific knowledge to make up stuff" seems to be engineers. 

When I ctrl+f-replace "inventors" with "engineers" in my head, I personally see your career path theory making more sense given that engineers do have a career path, which is mostly to become well-degreed technicians, financiers, or tenture-track-warriors. They ought to becoming inventors, but the existing paths divert them.

Corporate research is largely not as ambitious and long-term as it used to be.

A large part of this may be that there is increasing pressure on CEOs to focus on short-term earnings at the expense of long term earnings. 

You might like Scientific Freedom by Donald Braben. It's a whole book about the problem of developing incentives for basic research.

Thank you, great essay.  Reminds me of parts of "The Age of Unreason" by Charles Handy, but that was from the 1990s guessing at what most working people would experience.  Curious if anyone thinks Handy's "portfolio professional" aligns with career paths for those hoping to invent.  This definitely hits on many experiences I have had in my career as I have moved to different organizations.  

I would say our engineering workshop is staffed by inventors. We need people to invent solutions to problems - they do it, usually in collaboration with the scientist that has a problem. I think this is a pretty common setup although not the model for lone inventor producing killer product that is patented and sold. I rather liked their solution for remote camera lens cleaner - (drone with super soaker. No public video sorry).

Alternative theory: most of the low-hanging fruit has been picked. In the information-technology side of things we see an enormous amount of innovation, often because it is possible to create new products with a garageband skeleton crew. In the physical realm, thing are simply much harder, much more specialized. 

Also, there is a negative feedback loop: since the world is much more specialized there are fewer people working in technical fields, and hence fewer people having the basis in technical know-how that might lead to chance-inventions [which I learned from you was a typical story during the early Industrial Revolution in Great Britain]. 

How would you respond to this take?

Any ideas on what may have caused companies to trend towards short-termism?

I think the most dire need in this area is something I call "open engineering".

If you create a physical invention then there is a difficult, but well-known career path: become CEO, raise money, do marketing, build the invention, sell it to customers.

But what about those who want to make an intellectual invention? Yes, yes, copyright exists, so you can sell copies of (e.g.) a piece of software, but when was the last time you saw someone successfully selling a new general purpose programming language? That almost never happens, and there's a very good reason why: because learning a new programming language requires time and effort, which for most people is not worthwhile unless there is an employer out there who wants to hire you to use that language. A language costs a lot of time even if it's free! Good luck adding a monetary cost on top of that: you're saying that after you've spent all this time learning the language, you can't necessarily use it until you convince your employer to pay for it? Uh-huh...

But normally, employers don't want to use an obscure language, even a free one, and indeed, I've heard various examples of crappy proprietary languages built in-house at companies that the employees of those companies like to groan and gripe about. There's no theoretical reason that a proprietary language couldn't be good, but in practice they're not, because there is virtually no market for them, so they tend to be underfunded even in cases where they exist at all. People who learn brand new languages, at first, tend to be people who do it for fun, not for profit. Therefore if the inventor charges money for it, these early adopters probably won't be interested.

By producing an open-source language, the inventor forfeits any practical path to profitability, but it's much easier to get early adopters. More than that, it provides a path to creating a crucial network effect: employers, very slowly at first, become interested in using the language if it has major benefits, which slowly encourages developers to learn the language, and as the number of developers who know the language increases, to too does the number of employers willing to use it, in a virtuous cycle.

As evidence of this, I cite its opposite: Mathematica, a very cool, capable and versatile programming platform, which however is proprietary, and I think this is why it has never taken off among software developers. You buy it if you really need what it offers; otherwise you ignore it and use JavaScript (all the while griping about its stupid design decisions). Mathematica has something no other language has: an huge and unified library of mathematical functionality. It's also fundamentally graphical in nature—another rare feature. Clearly this wasn't cheap to produce, so they recoup their costs by charging money, which, however, may have reduced Mathematica's popularity by orders of magnitude.

No only that, but by going open-source the inventor gains something else that any EA like myself salivates over: the ability to improve the world in a real and significant way. If my language is free, anyone can use it, and that includes schoolchildren and developing-world residents who normally can't afford to buy software. That's a real win on the "warm fuzzies" side of things. People like me care more about improving the world than making money, so producing expensive proprietary software has no appeal except for the purpose of making money. I do need money, sure, absolutely, very much so. But I know that making a proprietary programming language or library is a slow path to riches, so instead I am building software for the oil and gas industry which, if not for the money, I would prefer not to support! (I'm in Calgary, Canada's oil & gas hub, so that's a very common kind of job here, so I took that job and spend some of my paycheck on clean energy advocacy.)

Another key foundational component of software—arguably more important than programming languages—is libraries, units of code such as algorithms, data structures and protocols, that are designed to be re-used by an unlimited number of people and companies. Here too, the proprietary route is possible, but usually you have to charge very high prices to make it worthwhile, since most developers prefer to use an inferior open-source product if available, and you'll probably have to spend a lot on marketing for people to even discover your product.

Case in point, my company was looking for a Simplex algorithm for C# — a algorithm taught in many university courses, but not an easy algorithm to implement correctly and robustly. There were expensive commercial products available that supported it, but we're a tiny company with no customers, so we decided to go the open source route. Guess what, though? There are no good Simplex implementations for C#! To get Simplex in C#, I extracted the Simplex algorithm from the massive Apache Math library for Java. This was not easy, because that algorithm is embedded in a complex object-oriented framework and the transitive closure of its dependencies was perhaps dozens of times larger than the core algorithm.

I got the job done, so now we have a "free" Simplex algorithm in C# that just required a few days of engineering work, but have we released it as open source? No, we have not. Why? Because although Simplex is useful in a wide variety of industries and could significantly benefit the world, it could potentially also benefit a competitor. I think this phenomenon happens very widely in the software industry and is certainly an impediment to progress. Now, my company has good people who support open source, so they are willing to release the code after we have a few customers. But we're not planning to support or improve the software, it'll just be a code drop. Look at the wasteful duplication here: Simplex for C# already exists, but we didn't want to  pay for it, so we built our own. This is a waste of human labor that we accept because we have no better system. (Imagine if, instead, engineers spent more time creating new things instead.)

Another example from this year is a library called SyncLib that I built for a company before being laid off due to the Covid downturn. SyncLib is a unique serialization library that I have demonstrated can easily cut your serialization code size in half while making it less error-prone, supporting multiple data formats, and maybe even improving performance. It's useful in any industry and great for hobbyists too.

They refused to give me any of the source code of SyncLib, including even an interface file that would remind me how it worked. Why? Was it because competitors might benefit? No. It was because (under their own open-source policy) they would have someone review, publish and maintain the code, and they didn't want to spend money on that just for some forgotten ex-employee.

So in order to build an open-source version I had to do it all from scratch, again. The new version is much more ambitious, and of course, free. But how can I afford to build an open-source product? Simple: I have a deal with my employer in which they pay me 20% less money in exchange for 20% open-source time (because I get burned out if I have to do it all on weekends).

This, Jason, is what I'd like you to advocate: there should be funding for open engineering. I don't care if the pay is lousy, either. Low pay — and I mean low, as in minimum wage or thereabouts — has a huge advantage for anyone who wants to improve the world. The advantage is that whoever supplies the funds can be less picky about whose applications for funding they can accept. Paying $12/hour, they can fund three times as many projects as they could at $36/hr. Hopefully that means I don't have to prove to some conventionally-minded grant reviewer that my radical idea is a good one before I can get any funding for it.

Support Open Engineering! And by the way, it's not just for software. No doubt there are people out there who would like to write free textbooks, or help design open RISC-V microprocessors, or blueprints for gadgets you can mostly print at home on your 3D printer, etc.