S-Curves for Trend Forecasting

post by mr-hire · 2019-01-23T18:17:56.436Z · score: 100 (37 votes) · LW · GW · 14 comments

Contents

  Basics
  Diffusion S-Curves
  Evolution S-Curves
  S-Curves vs. Exponential Growth
  S-Curves Patterns
    Diffusion Patterns
      Diffusion of Innovation
      The Chasm
    Evolution Patterns
      Wardley Evolution
      Simplicity - Complexity - Simplicity
      Disruptive Innovation
      Gartner Hype Cycle
      Windermere Buying Hierarchy
  Conclusion
None
14 comments

Basics

This is an S-curve.

The S-curve is a fundamental pattern that exists in many systems that have positive feedback loops and constraints. The curve speeds up due to the positive feedback loop, then slows down due to the constraints.

When the constraint is broken, the positive feedback loop ramps back up, until it hits another constraint.

Diffusion S-Curves

The S-curve pattern is quite common in the spread of ideas, practices, and technologies, although it rarely looks quite as pretty. The example below shows "diffusion s-curves" - How a technology spreads through a population (in this case US households

The positive feedback loop in this case is word of mouth, and the constraints represent fundamental barriers to certain market segments or growth such as simplicity, usability, scalability, price, etc.

This creates smaller s-curves around adoption among specific market segments, and larger s-curves that represent the overall market penetration of the idea, practice, or technology.

Evolution S-Curves

In addition to Diffusion S-curves in technology, ideas, and practices, there are Evolution S-Curves. These represent the increase in the traits of these ideas that make them usable in more situations and desirable for more people. When you break through a constraint in one of these properties through innovation, this can often coincide with "unlocking" a new diffusion curve by opening up a new market that wouldn't previously have used your technology or idea.

In this case the positive feedback loop is the increased understanding and expertise that comes from diffusion of a new innovation in your idea or technology, and the constraint represents fundamental assumptions in the idea, practice, or technology that must be changed through another innovation to make the idea, practice, or technology more desirable.

In the example below the desirable property is hardware speed. Fundamental leaps are made to break through a speed constraint, and then iterated on through the positive feedback loop of information and expertise increasing from adoption. This hits diminishing returns as the new innovation is optimized, and then a new fundamental innovation is needed to overcome the next constraint.

S-Curves vs. Exponential Growth

Sometimes, people get confused and call S-curves exponential growth. This isn't necessarily wrong but it can confuse their thinking. They forget that constraints exist and think that there will be exponential growth forever. When slowdowns happen, they think that it's the end of the growth - instead of considering that it may simply be another constraint and the start of another S-Curve. Knowledge of Overlapping S-Curves can help you model these situations in a more sophisticated way.

S-Curves Patterns

S-curves become quite useful when paired with an understanding of evolutionary patterns. They can allow you to see in a broad sense what's coming next for an idea, practice or technology. They can prevent surprises and give you a tool to stay ahead of changes.

There are patterns that exist for both diffusion and evolution S-curves.

Diffusion Patterns

Diffusion patterns describe common themes that happen as trends diffuse through a population. They apply on the micro-level to individual population-segments, and on a macro-level to the overall population.

Diffusion of Innovation

The diffusion of innovation describes 5 separate stages of a diffusion curve: Innovators, Early Adopters,Early Majority, Late Majority, and Laggards. By understanding the traits of each of these groups, you can get a broad idea of what to expect, and how to slow or speed up adoption.

The Chasm

The Chasm describes a common constraint that occurs in a market segment between "early adopters" - who are willing to put up with a lot, and "early majority", who expect a lot. There is often a number of evolutionary constraints that must be broken through to bridge this single diffusion constraint and many new ideas, practices, and technologies get stuck in the chasm for that reason.

Evolution Patterns

Evolution patterns describe common ways that innovations evolve over time to become increasingly desirable. They apply on the micro-level to individual innovations within a trend, and on a macro-level to the evolution of trend as a whole.

Wardley Evolution

Innovations tend to go through four stages - the initial prototype, custom built versions, productized versions that compete, than comoditized versions that are all basically the same. By understanding where you are, you can understand the type of competition likely to happen, the types of processes likely to yield improvements, and large changes that will be needed to stick with the market.

Simplicity - Complexity - Simplicity

Innovations tend to start out relatively simple as a new approach to a problem. They become increasingly complex to cover more use cases and be more robust, and then become simple again as refinements are made and they're distilled to their essence.

Disruptive Innovation

Sometimes, innovations overshoot the mainstream populations needs on a particular dimension in order to be powerful for a particularly lucrative part of the population. In this case, these innovations or often overtaken by subsequent innovations that lower the performance on that dimension in order to raise it on other dimensions (example: Lower flexibility of a software product but raise the simplicity), these innovations can then "disrupt" the original innovation.

From the perspective a current innovation, the disruptive innovation appears to start below it in the s-curve, but it's able to gain adoption because the particular performance feature of that innovation is already higher than the market needs, and the new product competes on a different performance feature that is not even a target o

(image of how that plays out with S-Curves)

Gartner Hype Cycle

The Gartner Hype Cycle describes a particular way that the media over-inflates people's expectations of new innovations in comparison to how evolved they actually are for a particular market segment's needs.

Windermere Buying Hierarchy

The Windermere Buying Hierarchy describes four different improvement focuses that an innovation optimizes over time. First, it's trying to solve for functionality, then reliability, then convenience, and finally price. This loosely maps to the stages of Wardley Evolution.

(image here)

Conclusion

S-curves and s-curve patterns are a useful tool for quickly analyzing systems, particularly when looking at diffusion of trends and evolution of innovations. They can heuristically identify solutions and probabilities that would otherwise be quite time consuming to figure out using something like a full system or functional analysis.

Hopefully you find this tool useful in your quest to understand all the things.

14 comments

Comments sorted by top scores.

comment by Raemon · 2019-03-01T20:12:14.291Z · score: 18 (7 votes) · LW · GW

This post reminded me of another great post that delves more deeply into the S-curve phenomenon, Invisible Asymptotes. It's written by someone who was involved with Amazon in their early days, as they found their growth slowing and needed to predict what was the next bottleneck to their growth as they went through the S curve cycle.

For me, in strategic planning, the question [ was to flush out what I call the invisible asymptote: a ceiling that our growth curve would bump its head against if we continued down our current path. It's an important concept to understand for many people in a company, whether a CEO, a product person, or, as I was back then, a planner in finance.
Amazon's invisible asymptote
Fortunately for Amazon, and perhaps critical to much of its growth over the years, perhaps the single most important asymptote was one we identified very early on. Where our growth would flatten if we did not change our path was, in large part, due to this single factor.
We had two ways we were able to flush out this enemy. For people who did shop with us, we had, for some time, a pop-up survey that would appear right after you'd placed your order, at the end of the shopping cart process. It was a single question, asking why you didn't purchase more often from Amazon. For people who'd never shopped with Amazon, we had a third party firm conduct a market research survey where we'd ask those people why they did not shop from Amazon.
Both converged, without any ambiguity, on one factor. You don't even need to rewind to that time to remember what that factor is because I suspect it's the same asymptote governing e-commerce and many other related businesses today.
Shipping fees.
People hate paying for shipping. They despise it. It may sound banal, even self-evident, but understanding that was, I'm convinced, so critical to much of how we unlocked growth at Amazon over the years.
People don't just hate paying for shipping, they hate it to literally an irrational degree. We know this because our first attempt to address this was to show, in the shopping cart and checkout process, that even after paying shipping, customers were saving money over driving to their local bookstore to buy a book because, at the time, most Amazon customers did not have to pay sales tax. That wasn't even factoring in the cost of getting to the store, the depreciation costs on the car, and the value of their time.
People didn't care about this rational math. People, in general, are terrible at valuing their time, perhaps because for most people monetary compensation for one's time is so detached from the event of spending one's time. Most time we spend isn't like deliberate practice, with immediate feedback.
Wealthy people tend to receive a much more direct and immediate payoff for their time which is why they tend to be better about valuing it. This is why the first thing that most ultra-wealthy people I know do upon becoming ultra-wealthy is to hire a driver and start to fly private. For most normal people, the opportunity cost of their time is far more difficult to ascertain moment to moment.
You can't imagine what a relief it is to have a single overarching obstacle to focus on as a product person. It's the same for anyone trying to solve a problem. Half the comfort of diets that promise huge weight loss in exchange for cutting out sugar or carbs or whatever is feeling like there's a really simple solution or answer to a hitherto intractable, multi-dimensional problem.
Solving people's distaste for paying shipping fees became a multi-year effort at Amazon. Our next crack at this was Super Saver Shipping: if you placed an order of $25 or more of qualified items, which included mostly products in stock at Amazon, you'd receive free standard shipping.
The problem with this program, of course, was that it caused customers to reduce their order frequency, waiting until their orders qualified for the free shipping. In select cases, forcing customers to minimize consumption of your product-service is the right long-term strategy, but this wasn't one of those.
That brings us to Amazon Prime. This is a good time to point out that shipping physical goods isn't free. Again, self-evident, but it meant that modeling Amazon Prime could lead to widely diverging financial outcomes depending on what you thought it would do to the demand curve and average order composition.
To his credit, Jeff decided to forego testing and just go for it. It's not so uncommon in technology to focus on growth to the exclusion of all other things and then solve for monetization in the long run, but it's easier to do so for a social network than a retail business with real unit economics. The more you sell, the more you lose is not and has never been a sustainable business model (people confuse this for Amazon's business model all the time, and still do, which ¯\_(ツ)_/¯).
The rest, of course, is history. Or at least near-term history. It turns out that you can have people pre-pay for shipping through a program like Prime and they're incredibly happy to make the trade. And yes, on some orders, and for some customers, the financial trade may be a lossy one for the business, but on net, the dramatic shift in the demand curve is stunning and game-changing.

The article is quite long (this lengthy excerpt is, like, just the prologue). The post covers a number of interesting areas.

(Overall this blog actually makes me feel the most like reading a Scott Alexander post, i.e. most posts are quite long, covering multiple lenses through which to look at an interesting problem)

comment by mr-hire · 2019-03-02T10:36:46.279Z · score: 2 (2 votes) · LW · GW

I wanted to work in this idea of getting ahead of constraints but it didn't really seem to fit anywhere.

I agree Eugene is among my top 3 business strategists to follow online and I learned a lot about how to think about strategy by reading his blog.

Idea - a thread for the best blogs to follow on every subject.

comment by elityre · 2019-01-24T18:40:19.680Z · score: 11 (6 votes) · LW · GW

I'm glad to have read this. In particular:

Sometimes, people get confused and call S-curves exponential growth. This isn't necessarily wrong but it can confuse their thinking. They forget that constraints exist and think that there will be exponential growth forever. When slowdowns happen, they think that it's the end of the growth - instead of considering that it may simply be another constraint and the start of another S-Curve.

This is obvious in hindsight, but I hadn't put my finger on it.

Thank you!

comment by elityre · 2019-04-25T15:05:36.757Z · score: 6 (3 votes) · LW · GW

Intriguingly, after coming back to this comment after only 3 months, my feeling is something like "That seems pretty obvious. It's weird that that seemed like a new insight to me."

So I guess you actually taught me something in a way that stuck.

Thanks again.

comment by jmh · 2019-03-11T17:39:09.761Z · score: 2 (2 votes) · LW · GW

Plus one on that point.

In once sense though it seems a rejection of, what I will call, the S-Curve mentality. That would be the thinking all growth always plateaus (and it seems that is a dominant view in terms of economic growth there -- developing economies can grow faster then developed economies so all these fast growers are doomed to the fate of Japan, Europe or the USA). That thinking can lead to acceptance rather than effort to overcome some current limitation/constraint.

comment by Jameson Quinn (jameson-quinn) · 2019-03-05T01:11:05.461Z · score: 4 (3 votes) · LW · GW

How do I know when some trend isn't made of S-curves? How do S-curves help me make predictions, or, alternately, tell me when I shouldn't try predicting? Is this falsifiable?

comment by mr-hire · 2019-03-05T12:22:04.228Z · score: 5 (4 votes) · LW · GW

> Is this falsifiable?

Innovation research is notoriously hard to falsify and subject to just-so stories and post-hoc justifications.

One of the things I find compelling about S-curves is just how frequently they show up in innovation research coming from different angles and using different methodologies.

Some examples:

  • Everett rogers is a communication professor trying to figure out how ideas spread. So he finds measurements for ownership of different technologies like television and radio throughout society. Finds S-curves.
  • Clayton Christensen is interested in how new firms overtake established firms in the market. Decides to study the transistor market because there's easy measurements and it moves quickly. Finds S-curves.
  • Carlotta Perez is interested in broad shifts in society and how new innovations effect the social context. She maps out these large shifts using historical records. Finds S-curves.
  • Genrich Altshuller is interested in how engineers create novel inventions. So he pores through thousands of patents, looks for the ones that show real inventiveness, and tries to find patterns. Finds S-curves.
  • Simon Wardley is interested in the stages that software goes through as it becomes commodotized. Takes recent tech innovations that were commodotized and categorizes the news stories about them, then plots their frequency. Finds S-curves

> How do S-curves help me make predictions, or, alternately, tell me when I shouldn't try predicting?

By understanding the separate patterns, they can give you an idea of the most likely future of different technologies. For instance, here's a question on LW that I was able to better understand and predict because of my understanding of S-curves and how innovations stack. [LW · GW]

> How do I know when some trend isn't made of S-curves?

I think understanding how to work with fake frameworks [LW · GW] is a key skill here. Something like S-curves isn't used in a proof to get to the right answer. Rather, you can use it as evidence pointing you towards certain conclusions. You know that they tend to apply in an environment with self-reinforcing positive feedback loops and constraints on those feedback loops. You know they tend to apply for diffusion and innovation. When things have more of these features, you can expect them to be more useful. When things have less of these features, you can expect them to be less useful. By holding up a situation to lots of your fake frameworks, and seeing how much each applies, you can "run the Bayesian Gauntlet" and decide how much probability mass to put on different predictions.

comment by Mark Webb (mark-webb) · 2019-04-25T13:28:57.390Z · score: 1 (1 votes) · LW · GW

I think you'll always be working in S-curves if you're in a finite system. The trick is to be able to detect the rate-limiting factor. That's the factor that marks the inflection point between exponential growth and the beginning of the slowdown. For classic examples like bacterial growth that might be nutrients, space, elimination of waste, etc.

The hard part is determining whether you've considered all the rate-limiting factors involved. Going back to bacterial growth, if you think food is the rate-limiting factor and you predict your culture will continue to grow for six hours, you might be surprised when you hit an inflection point after three hours because waste products start killing bacteria off. This same principle can be applied in technology and elsewhere, where people often aren't even looking for rate-limiting factors and appear to assume exponential growth in a finite system.

comment by mr-hire · 2019-04-25T19:03:50.040Z · score: 1 (1 votes) · LW · GW

Agree, recognizing constraints before they limit you is key if you're looking to create growth (or alternatively, adding multiple constraints if you're looking to slow growth).

comment by Mark Webb (mark-webb) · 2019-04-26T06:25:23.354Z · score: 1 (1 votes) · LW · GW

If you want to slow growth, pick any limiting factor and apply pressure. One will do.

Sometimes a trend continues growing exponentially for a long time before bumping up against a limiting factor. The thing to remember about an S-curve is that if you plot it on a log scale the first half of the curve looks like a straight line all the way backward. That's because it's exponential growth at the beginning, so every new observation dwarfs all those that came before. Sometimes we spend a lot of time in exponential growth phase and people write articles about how it'll go on forever, and The Singularity, and whatnot. When you don't know the limiting factor, it's very tempting to fit your model to exponential growth, only to get burned later on.

comment by mr-hire · 2019-04-26T12:28:04.506Z · score: 1 (1 votes) · LW · GW

Agree, this misconception (and seeing it everywhere) is one of the things that made me wrote the article (particularly the part about "exponentional growth vs. s-curves).

The other side of it is when people think that trends are made of a single s-curve, and think that when growth is slowing down, that means that the trend is done forever, rather than simply the start of another s-curve when the constraint is defeated.

comment by Mark Webb (mark-webb) · 2019-04-26T20:54:26.929Z · score: 1 (1 votes) · LW · GW

Yes, I think it's an excellent article, especially the observation about constraints. If we can correctly identify which elements are constraining a system we have a path to return to exponential growth.

Still, we'll see articles lamenting that "despite how we've overcome Constraint X, growth hasn't returned." The world is multi-causal/multi-factoral, though. More than one factor can constrain growth. It is often an engineering problem, and focusing on the system as driven by rationally understandable forces is important. Otherwise the default seems to be to view trends as 'magical growth' and make illogical predictions based on that thinking.

In the case of growth in the computer hardware industry, where you have a veritable army of engineers focused on the problem, is it any wonder we continuously overcome constraints?

comment by Raemon · 2019-03-01T19:56:58.126Z · score: 3 (2 votes) · LW · GW

This was a great explanatory post, that distilled a lot of complex ideas into something short and accessible. I've curated it.

A minor thing that would have improved it is, for the various types of curves (especially ones named after people), it might have been nice to link to an existing canonical reference for that idea, that covers it in more depth or at least gives a sense of how other people have thought about it.

comment by Max Shtein (max-shtein) · 2019-05-08T18:40:46.988Z · score: 4 (3 votes) · LW · GW

Scalable systems all turn out to have these S-curve dynamics. It's roughly based on how networks operate, and other people have expounded upon this extensively – e.g. Geoffrey West, etc. Networks have certain identifiable elements, which can be projected onto specific situations for a deeper analysis. You can read a bit more about the System model and how it impacts technology innovation here: https://www.amazon.com/Scalable-Innovation-Eugene-Shteyn/dp/1466590971