Why do stocks go up?

Stock prices represent the market's best guess at a stock's future time-discounted price. Stock prices go up even in the absence of technological advancement because stocks are tied to the bond market [? · GW] via arbitrage.

But that is a complicated abstract idea. Let's examine things in simple Marxist terms.

A business is a machine that extracts rents from the proletariat. A stock is partial ownership of a business. Traditionally, the business extracted money and then distributed that money to shareholders in the form of dividends. This is called a dividend stock. In principle, a dividend stock does not go up in value. You just extract rents until the Revolution.

However, publicly-traded companies often reinvest profits into themselves instead of distributing revenues to shareholders as dividends. This is called a growth stock. If a company reinvests profits into itself then the total value of the company goes up because additional capital has been invested into it. Since you own an undiluted share of the company, the value of your stock goes up in equal proportion to the growth of the company.

The equity premium puzzle is still unsolved. The answer to your question is that nobody knows the answer. Stocks shouldn't have gone up historically, none of our current theories are capable of explaining why stocks did go up. Equivalently, stocks were massively underpriced over the last century or so and nobody knows why.

If you don't know why something was mispriced in the past, you should be very careful about asserting that it will or won't continue to be mispriced in the future.

Suppose at some point there is an announcement that in ten years Free Hardware Foundation will release a magical nanofactory that turns dirt into most things currently in the basket of goods used to calculate inflation. There is no doubt about truth of the announcement. No company directly profits from the machine, as it's free (libre) hardware.

There's some upheaval in the market, that eventually settles down. Yet real value of stock is predictably going to sharply go up after ten years, not (just) immediately, as that's when the basket of goods actually becomes cheaper.

One word : risk. There is uncertainty as to the company’s ability to provide that future revenue and as That uncertainty reduces, the stock price goes up.

If I can summarize your question as something like "can I beat the returns on an index fund by only investing in companies with new/useful technologies", I think you'll find this question is similar to "which version of the EMH is true", and you'll also find a lot of good discussions about this, for example here: https://www.themoneyillusion.com/are-there-any-good-arguments-against-the-emh

For the two stories presented, I would say Story 1 is trivially true and Story 2 is probably false, although it's not phrased super well (for example, is "advances in technology" company specific or general economic growth?). Trying to read between the lines, it seems like you're wondering something like "if my choice to invest is between two companies, and Company 1 has current cash flow of $1-million and future cash flow of $1-million (no growth), and Company 2 has current cash flow of zero but future cash flow of $1-trillion (lots of growth), should I always invest in Company 2?" And my answer is "no, unless you think weak EMH is true and there's a specific inefficiency that you can uncover through your research" ( and even then, your research might determine you should sell rather than buy Company 2).

To improve the framework, I suggest the following distinctions/assumptions:

• Some form of EMH (weak, semi strong, strong) is true (I tend to agree with Sumner that anti-EMH models are not useful).
• Value vs Price: Value is determined by things like future cash flows, risks, discounts and opportunity costs. Price is determined by supply and demand. I recommend Damodaran for discussions on this distinction, for example http://aswathdamodaran.blogspot.com/2020/03/a-viral-market-meltdown-iii-pricing-or.html
• There's a positive link/correlation between stock price and stock value in the long term. (In the short term price and value might diverge.)
• Real vs Nominal: Real value/price is determined by things like goods produced, services provided, technology/productivity growth. Nominal value/price is determined by the above but also the supply and demand of money, inflation, exchange rates. I recommend Sumner for discussions on this distinction, he blogs at Econlog and TheMoneyIllusion.
• Also, note that "technology" has a non-colloquial meaning under most economic models. For example, the Solow-Swan growth model says something like long term growth is determined by growth in the labor force and growth in "technology", where "technology" is basically anything that's not labor force and includes things like human capital/knowledge accumulation/diffusion and social/political institutions.

Using this framework, you could change your question to something like "assuming weak EMH is true, what sorts of public information about a company's new/useful technologies would allow me to value a company more accurately than the average investor". Then you could search for studies that try to answer this question or something similar.

The biggest reason stocks go up is pretty simple, in my view: a lot of very smart and hardworking people are working very hard to make stocks go up. In addition, lots of less smart and less hardworking people are also working to make stocks go up. In contrast, very few people are trying to make stocks go down.

While there are shortsellers, they are generally not trying to make stocks go down, i.e. by destroying value. Instead, shortsellers are simply saying that some companies are overvalued, and time and effort is better spent on other companies.

What do I mean by "make stocks go up"? I mean it in a fundamental sense: employees are trying to create value by making products better or adding new markets. Their rewards include stock compensation, bonuses, and promotions, so they are indeed incentivized to create value in whatever way possible. Meanwhile, financiers are trying to allocate capital (an abstract representation of time, effort, and value) in the best way, by optimizing society's effort. These people are also rewarded with capital gains, bonuses, and promotions.

Given that there is so much effort and incentive to fundamentally increase the value of stocks, why should the market stay flat or neutral? I'm sure someone can try to find some elaborate corner case why this isn't true, but mostly the reasons stocks wouldn't go up is if no one is trying (no incentive) or if someone is trying to destroy asset value (expropriation, violent revolution, etc.). If people are trying to build stuff that makes stock prices go up, and there isn't much effort to destroy them, it's pretty straightforward that they go up.

For what it's worth, this is basically Buffett's view. Also note, the comments about equity risk premium, valuation puzzles, and the efficient market hypothesis miss the point: that's only a matter of relative returns, not absolute returns. (If your question is about beating the market, then those points are more relevant, but it's not clear anyone can beat the market today in developed economies outside narrow pockets of inefficiency.) As long as you can build new stuff, you'll have a reason to invest, which demands a return, hence the positive returns we see. If there's less "stuff" to build, you can expect lower prospective returns because capital competes for each opportunity, resulting in lower rates and a huge premium on growth opportunities, which is what we observe today. 

One last way to think about it is in real terms. If you have a machine that can build more of itself, you will have more of those machines over time, which is a real return.

"Stock prices represent the market's best guess at a stock's future price."

But they are not the same as the market's best guess at its future price. If you have a raffle ticket that will, 100% for definite, win $100 when the raffle happens in 10 years time, the the market's best guess of its future price is $100, but nobody is going to buy it for $100, because $100 now is better than $100 in 10 years.

Whatever it is that people think the stock will be worth in the future, they will pay less than that for it now. (Because $100 in the future isn't as good as just having the money now). So even if it was a cosmic law of the universe that all companies become more productive over time, and everyone knew this to be true, the stocks in those companies would still go up over time, like the raffle ticket approaching the pay day.

Toy example:
1990 - Stocks in C cost $10. Everyone thinks they will be worth $20 by the year 2000, but 10 years is a reasonably long time to wait to double your money so these two things (the expectation of 20 in the future, and the reality of 10 now) coexist without contradiction.
2000 - Stocks in C now cost $20, as expected. People now think that by 2010 they will be worth $40.

Don't forget the Greater Fool theory (https://www.investopedia.com/terms/g/greaterfooltheory.asp).  Stocks go up because investors expect them to go up, and those investors' purchases actually drive the price up.  

There is some underlying truth in terms of dividends and stock buybacks (effectively a dividend, but paid by reverse-dilution), but the vast (VAST) majority of invested dollars are fully disconnected from any actual business outcomes, except by the publicity and expectations channel.

I think things become simpler when you look at the sum of all stocks, versus particular ones. Then, you only need to consider the market cap of the entire stock market, and what makes it change over time.

The economy is much bigger than the stock market. Money flows from small companies to larger one as the economy consolidates -- since the former are more likely to be publicly traded than the latter, that makes the market become bigger.

It's easier to invest in the stock market now than in the past. Since it's accessible to more people, then more people's money can be put in it. So, the market cap goes up.

Finally, as inequality increases, more of a fraction of wealth is disposable, and therefore can be invested. That makes the market grow as well.

I'm sure there are many other reasons along these lines.

Here's some motions toward an answer. I'll consider an informally specified market model, as opposed to a real market. Whether my reasoning applies in real life depends on how much real life resembles the model.

In particular, consider as model an efficient market. Assume the price of any stock X is precisely its expected utility according to all evidence available to the market. Then the only way for the price to go up is if new evidence arrives. This evidence could be the observation that the company associated with the stock continues to exist and produce income, that it continues to produce income, or that the income it produces is is going up.

Those bits of evidence remind me of the risk perspective. Sure, investors may believe that a company, if it continues to exist, will one day be worth more money than they could ever invest today. But if they think there's a 5% chance each year that the company stops existing, then this can severely limit the expected utility of owning the stock right now. (You might ask, "What if I assign a nonzero probability to the class of futures where the company exists forever and the utility of holding its stock grows without bound?" which makes the expected utility infinite, and makes me suspect that defining utility over infinite spans of time is tricky.)

I feel like the time-discounting hypothesis makes a lot of sense and is probably part of the truth. To make sense of it within my toy model, I'd have to look at what time-discounting actually means in terms of utility. A reasonable assumption within this idealized model seems to be that the "utility" of anything you possess is equal to the maximum expected utility of anything you could do with it / exchange it for, including exchanges over time. (This is like assuming perfect knowledge of everything your could do with your possessions. Utility can never go up under this assumption, similar to how no legal move can improve a chess position in the eyes of a perfect player.) This means the utility of owning a stock is somewhere between the utility of its buy price and its sell price, as expected. And the utility of 1 dollar right now is no less than the expected utility of a dollar in 2030, given that you could just hold on to it. Then time-discounting is just the fact that the utility of 1 dollar right now is no less than the expected utility of buying one dollar's worth of stock X and waiting a couple years. Let's assume the stocks grow exponentially in price. That is to say, though neither the stocks nor the money increases in utility, stocks can be exchanged for more and more money over time. It seems converting money into stocks avoids futures where we lose utility. So how much should we pay for one stock? This is just determined by the ratio of the utility of the stock to the utility of a dollar. So the question becomes, why does money have any utility at all, if it is expected to fall in utility compared to stocks? And this must be because it can be exchanged for something of intrinsic utility, such as the enjoyment derived from eating a pizza. But why would someone sell you a pizza, knowing your money will decrease in utility? Because the pizza will decrease in utility even faster unless someone eats it, and the seller has too many pizzas to eat, or wants to buy other food for themself. (Plus, the money is backed by banks / governments / other systems.)

So if the average price of stocks tends to increase year by year, why are they not worth infinite money to begin with? Here's another perspective. How is the average price calculated? Presumably we're only averaging over stocks from active companies, ones that have not ceased to exist due to bankruptcy or the like. However, when evaluating the expected utility of a stock, we are averaging over all possibilities, including the possibilities where the company goes bankrupt. There is a nonzero chance that the company of a stock will cease to exist (unpredictably). So if you Kelly bet, you should not invest all your money into such a stock. As a consequence, successively lower prices are required to get you to buy each additional stock of the company.

Reflection: I imagine these concepts match with ideas from economic theory in quite a few places. I have a mathematical background myself, probably making the phrasing of this answer unusual. I'm not so sure about this whole informal, under-specified model I just made. It seems like the kind of thing that easily leads to pseudoscience, while at the same time playing around with inexact rules seems useful in early stages of getting less confused, as a sort of intuition pump. (Making the rules super strict immediately could get you stuck.)

I don't know the full answer to why stocks go up, but I have a partial answer based on risk. Imagine there are only 2 products available in the market:

(1) A US government bond that pays $100 in 1 year.

(2) Ownership in a company that will dissolve in 1 year, and at the end either return to the owner $95 or $105 with equal probability. 

Note that both have the same expected return in 1 year, $100. But people will prefer to buy the first product compared to the  second one, since the first one is risk free. Say the current 1-year interest rate for risk free returns is 1%, then people will pay $99 for the first product. But since the second product is less desirable, they might only pay $98 for it. So the second product has greater expected profit, since you paid $98 for $100 of expected returns. If you only invest in products like (2), then in the long run you'll make more money in expectation compared to investing in risk-free assets like (1).