Alpha

post by Erich_Grunewald · 2023-07-01T16:05:55.940Z · LW · GW · 2 comments

This is a link post for https://www.erichgrunewald.com/posts/alpha/

Contents

  Summary
  Form-Plan
  Murnane's Dream Was Benter's Reality
  Alpha, Seen Up Close
  Alpha, Seen from a Distance
  Tentative, Generalised Ways of Finding Alpha
  References
None
2 comments

Summary

In the first half of this essay, I recount two anecdotes. (The impatient reader can skip these.) First, Gerald Murnane tries, around Melbourne in the 1950s, to find a system that'll make him money betting on horse races. Then, Bill Benter, in Hong Kong in the 1990s, comes up with a system for picking horses that makes him nearly $1B.

In the second half of this essay, I discuss alpha:

Form-Plan

In the long-ago 1960s, I knew a man who claimed to have been helped through a troubled period by his faith in psychoanalysis. When I myself was going through such a period, he urged me to read a certain huge book on the subject. I've forgotten the title but I recall the author's name, which was in gilt letters on the dark-green spine: Otto Fenichel. I read several chapters but I recall today only two short passages. One passage described the symptoms of a man with obsessive-compulsive disorder. He could never walk more than a few paces forward without obeying an urge to look behind him for any beetle that was lying helpless on its back and needing to be set upright. The other passage was the opening sentence of a section on gambling. According to the learned author, the gambler gambles in order to learn whether or not God has forgiven him for his masturbation.

Those are the opening sentences of the chapter, not on insect welfare, but on betting systems in Gerald Murnane's fairly Australian autobiography Something for the Pain (Murnane 2015, 89--100). Murnane goes on to write that, had the psychoanalysts "ever learned how much time and effort I've put into my search for a reliable and profitable betting system, they could only have concluded that I was either the all-time champion Onanist or, at least, the one of all the practitioners of the ancient art who felt the most guilty about it".

Most punters probably have more in common with the beetle sympathiser than with the uneasy sinner: they're driven more by compulsion than scrupulosity. And yes, Murnane's motivation was (he reports), in his later years, simply the pleasure of discovery, and in his early years, more akin to that of the FIRE crowd -- to be able to "rent a comfortable flat in Dandenong Road, Armadale; to own a small car; to join a middle-level golf club; and to put together a library of a few hundred volumes of fiction and poetry, along with a select collection of long-playing records".

During those early years -- this was in the late 1950s -- Murnane began to notice advertisements for a betting system called Form-Plan. Murnane had begun his life-long pursuit of profitable racing systems in 1952 when, at the age of 13, he began to search the Sporting Globe for patterns in the past form of winners; these efforts naturally worked perfectly for historical races, but far from perfectly on new races (this was a decade or two before the term overfitting was coined). Form-Plan claimed to select 50% winners and 75% placings; it was lavishly marketed and -- this was unusual for an advertised betting system -- its ads prominently featured its creator's name, address and photograph likeness. It wasn't the only betting system to be advertised in the Sporting Globe at that time, but it was the only one that lasted longer than a year.

Of course Murnane wondered (he reports) why someone who'd discovered the secrets of racing would advertise their discovery in the Sporting Globe. He worried not only that Form-Plan might be fraudulent, but also that, even if it worked initially, its effectiveness would be short-lived now that it had been publicised across the state of Victoria. Another reason to worry about Form-Plan was its seeming to select very few horses, meaning punters would need to bet large sums to earn a living from it (if it even worked).

Still, Form-Plan caused a stir in Murnane's racing circles. After over a year of seeing it advertised, and together with a colleague, Murnane paid the 10 pounds (the equivalent of about 6-7 good-quality hardcover books, he offers) for a copy. The system provided a set of rules for selecting horses, for example choosing only so-called top weights (handicappers would assign weights to horses in accordance with their abilities so as to even out the field). It focused exclusively on two-year-old horses; this was one reason why it selected so few horses.

The first Form-Plan bet Murnane placed was 5 pounds (3-4 hardcover books) on a filly named Snowflower ("Pale blue, tartan sash"), who ended up losing narrowly at Caulfield Racecourse. Ten days later, he bet another 5 pounds on the filly that had beaten Snowflower, Faithful City ("Green, gold Maltese cross, striped sleeves and cap"), at four-to-one odds:

The favourite was Ritmar (White, purple stripes), a filly from Sydney ridden by Neville Sellwood. I wish I could remember who rode Faithful City. Ritmar could gallop only at top speed, and the rider of Faithful City, having seen this, held his mount together for a last run at the favourite. Ritmar led by two lengths around the turn. The straight at Moonee Valley is short but uphill. Ritmar began to tire and Faithful City to gain ground. [My colleague] and I were in the old South Hill reserve, almost head-on to the finish. We had no idea which filly had won. The judge studied the print of the photo finish for three or four minutes and then declared Faithful City the winner.

I have always maintained that a writer achieves nothing by trying to describe feelings; that feelings can only be suggested. I will therefore report only that my feelings, after Faithful City's narrow win, were mixed. I had in my possession the key to lifelong wealth, but so too had all those numerous unknown buyers of Mr Maclean's method. How long would it be before half the population of Melbourne heard about Form-Plan and flocked to the races to get their share of the easy money?

The next bet again was on Faithful City, who won at about two-to-one odds. Murnane bet 8 pounds (5-6 hardcover books) and profited the equivalent of ~4x his weekly allowance as student teacher. But soon thereafter the system began to make a loss, and after about a year of Form-Plan betting Murnane gave up on it:

I don't recall when the ads for the system disappeared from the Sporting Globe, but I recall my receiving a newsletter from Mr Maclean at some time in 1959. He had decided to change some of the rules of his method. We were no longer to back horses starting from wide barriers or on days when the track was heavy. Needless to say, these rules would have prevented me from backing several losers during the months past. The last message that I received from Mr Maclean was another newsletter. He had devised a completely new method of systematic betting. Results for the past few years were outstanding. Persons who had previously purchased Form-Plan could buy the new system at a discounted price. I decided that the man was incorrigible and a rogue, but that was a few years before I read Otto Fenichel's book. Now I incline to the belief that poor Mr Maclean was truly desperate to learn how he stood with God.

Murnane's Dream Was Benter's Reality

To the best of my knowledge, Gerald Murnane never found his winning system. But he was no Don Quixote. Others succeeded where he did not.

A decade and a half after Faithful City won the race at Moonee Valley, a horse racing columnist for the Washington Post named Andrew Beyer put into practice his own betting system:

In the early 1970s -- armed with sharpened pencils, a mini calculator, large sheets of paper, tall stacks of the Daily Racing Form and a bottle of Jack Daniel's -- Beyer performed his painstaking work. By carefully and thoroughly accounting for the biases of tracks and adjusting horses' clocked times accordingly, he collapsed the complications down into a single number: a speed figure. Speed figures allowed for reliable comparisons between California horses and Florida horses, between five furlongs and six, between a lowly claiming race and a big-money graded stakes. "Speed figures clarified mysteries, subtleties and apparent contradictions of the sport that I had always thought were beyond human understanding", Beyer wrote in 1975.

By then, Beyer was already a successful gambler, in large part because his private calculations gave him a healthy edge. His books' publication educated the market and eroded his advantage. Beyer accepted this trade-off because he wanted to be a published author.

Later on, in Hong Kong in the 1990s, Bill Benter, formerly an advantage player in Las Vegas, earned nearly $1B betting on horses. (Hong Kong had a vibrant, high-liquidity betting market for horse racing at the time.) The way he did it -- inspired by Brecher (1980) and Bolton and Chapman (1986) -- was by programming and continuously refining a computer model which at its most complex regressed >100 variables and comprised thousands of lines of code (Benter 2008). Benter's crowning glory was when he, on 3 November 2001, won the Triple Trio (worth ~$13M) by placing a bet correctly calling the top three horses (in any order) in three different races; the story goes (and here we should exercise scepticism) that he chose not to collect the winnings, instead letting the Hong Kong Jockey Club do with them what it always does with unclaimed winnings: give them to charity.

Like Murnane fearing Form-Plan's spread throughout Victoria, Benter too speculated that his edge would disappear as others adopt his methods (Benter 2008):

Even if the racing is honest, if the general public skill level is high, or if some well financed minority is skillful, then the relative advantage obtainable will be less. [...] [There is] no guarantee that the future will be as profitable as past simulations might indicate. The public may become more skillful, or the dishonesty of the races may increase, or another computer handicapper[1] may start playing at the same time.

These days, there are entire professional gambling operations devoted to betting on horses: "Collectively known as computer-assisted wagerers, or CAWs, they are largely anonymous. These sophisticated bettors use horseracing data to the extreme, employing algorithms, research staff and sweetheart deals to enrich themselves. In recent years, they have increased their capital and their wagering to unprecedented heights, accounting for as much as one-third of the money bet nationwide." In 2021, a single CAW syndicate represented ~30% of bets placed on Californian tracks, but of course they'd make no money without a sizeable proportion of amateur punters.

Alpha, Seen Up Close

In finance, alpha refers to (a) excess returns earned on an investment above the market as a whole when adjusted for risk[2], (b) an investor's ability to beat the market and (sometimes) (c) a strategy or resource that repeatedly and consistently generates excess returns. The boundaries between these seem fuzzy; I've heard people use "alpha" to refer to information, models, software, talents and securities among other things.

Outside finance, alpha refers by metaphor to someone's ability to beat their competitors: in other words, an edge, a particularly successful tactic, being ahead of the curve, as the expression goes. It's what most sports dynasties had, most groundbreaking artists and most farseeing scientists. It's what Gerald Murnane sought.[3] It's what Bill Benter found. It's not success, but it can be converted to it. It's the thing that Taleb (2007) describes:

Think about the "secret recipe" to making a killing in the restaurant business. If it were known and obvious, then someone next door would have already come up with the idea and it would have become generic. The next killing in the restaurant industry needs to be an idea that is not easily conceived of by the current population of restaurateurs. It has to be at some distance from expectations. The more unexpected the success of such a venture, the smaller the number of competitors, and the more successful the entrepreneur who implements the idea. The same applies to the shoe and the book businesses -- or any kind of entrepreneurship. The same applies to scientific theories -- nobody has interest in listening to trivialities. The payoff of a human venture is, in general, inversely proportional to what it is expected to be.

Aiming at the unexpectedly good usually involves some amount of risk. There's a widespread assumption, in finance at least, that in order to get expected returns greater than the broader market you must tolerate higher risk. On that view, the only way to get higher returns is to increase risk, and the only way to reduce risk is to accept lower returns. Alpha exists only where that assumption is false. Falkenstein (2009):

The key is that risk taking and risk tolerance is not like physical courage, the ability to withstand physical pain, but rather like intellectual courage, the ability to withstand ridicule. Physical courage is something most people agree on, and invites empathy, as when we cringe when watching someone crash his bike. In contrast, intellectual courage as it is sometimes practiced is usually seen as foolishness because risk is a deviation from what everyone else -- the wisdom of crowds, the vast majority of smart people -- is doing, and invites ridicule that is pointed and very hurtful.

Risk means exposure to possible negative consequences. When I "risk doing X", I expose myself to possible negative consequences of doing X. When I "put Y at risk", I expose Y to such consequences. As lsusr writes [LW(p) · GW(p)], "Alpha is a bet against your society's beliefs." And finding alpha does at least sometimes imply taking risks, though taking risks doesn't necessarily mean you find alpha: sometimes you pay a premium for risk, as illustrated by the gambler and the casino.

Richard Hamming, mathematician and computer scientist, pointed at courage as one of the virtues a researcher needs in order to attain greatness. But he seemed to think it was important less because pursuing hard problems was risky, and more because it required a certain amount of self-confidence, or failing that the strength of will to push through anyway. "Once you get your courage up and believe that you can do important problems, then you can. If you think you can't, almost surely you are not going to."[4]

Alpha, Seen from a Distance

Economists say we can make inferences about people's beliefs by observing their behaviour. If so, plenty of people believe there's alpha in financial markets. But the efficient-market hypothesis, or at least the strong version of it, says markets perfectly and immediately reflect all available information, meaning investors can't consistently beat the market using public data. What gives?

Markets are made up of people, and efficiency is not binary. When asset prices change as new information comes to light, that is because people make trades based on that information. Some of those people will make the right trades, and profit. The question is whether they can do that consistently. Austin Vernon writes:

One of the strange corollaries to the Efficient Market Hypothesis is that you can't tell if a market is efficient. But there is reason to believe America's public markets are very efficient. Over long periods, close to zero active managers can beat the market.

Renaissance Technologies is one of the most famous hedge funds. They employ some of the best mathematicians and scientists in the world to do quantitative modeling and predict stock prices. They are known for their Medallion Fund, which earns gobsmacking returns. But the fund has no outside investors, and it returns capital instead of compounding it. The fund's strategy focuses on short-term and events-based trading, not long-term analysis.

Renaissance's public funds have returns that look like many other hedge funds. Unremarkable even with all the mad scientists.

Hundreds of the best math minds in the world can only find a few billion dollars a year of inefficiency in market data. All evidence points to opportunities that are small and fleeting.

Renaissance likely is able to consistently find alpha, but a lot of what looks like alpha is probably only happenstance or survivorship bias. What would you expect to see if, each year, every hedge fund had an equal chance of making a profit and a loss? If you simulate 1K hedge funds, each of which has an annual profit sampled from a normal distribution around 0% (90% CI: -40% to +40%), you get the results shown below. In particular, the plot shows the returns[5] for these funds over time (compared with a risk-free return of +2% to +4%). Note the log scale. After 20 years, the probability that a fund outperforms the risk-free return is 16%; after 40, 7%; and after 60 years, 5%. Despite a pure coin-flipping strategy, 50 of the 1K funds will still beat the risk-free return six decades out.

img

How do you distinguish alpha from noise? When a young Jesse Livermore -- one of the most famous stock traders ever to live -- brought home $1K, his mother disapproved of his venture, dismissing it as "gambling". Livermore went on to make millions, declare bankruptcy, make millions again, declare bankruptcy a second time, then to net $100M with massive short positions during the 1929 Wall Street Crash, and in 1934 to declare bankruptcy a third time.[6] Livermore had a strategy for finding alpha -- he'd focus on historical price and volume data to look for trends and reversals, an early form of what's now called technical analysis -- and still did not consistently win out.[7]

Livermore (1940) wrote: "There are times when money can be made investing and speculating in stocks, but money cannot consistently be made trading every day or every week during the year. Only the foolhardy will try it. It just is not in the cards and cannot be done."

The stock market is still more efficient today, and alpha is rarer and more fleeting.

In other domains, alpha is everywhere. Take an ancient forest. The forest holds plants, fruit, berries, mushrooms, animals. It's also home to predators -- wolves, bears, lynxes, humans. Most predators struggle to survive. Sometimes an individual or species finds a winning advantage, but more often they're stuck in a population cycle (fun case studies for students of Lotka-Volterra equations). But human hunters can stroll in and, in a merry afternoon, find and kill enough game to feed themselves for days. Human culture is a constant, inexhaustible source of alpha when humans compete with the other animals.

There are domains -- not always the domains we're interested in -- where alpha is abundant, and domains -- often the more valuable ones -- where it is scarce. As a general rule, the more valuable the object, the harder it is to find alpha. There are exceptions: for example, goods in altruistic markets are often more valuable for society than you'd expect from the number of people who seek them, simply because people are a mite selfish and tend to seek goods that they value for themselves over goods that are valuable for society as a whole.[8] These exceptions are important, as we shall see.

Tentative, Generalised Ways of Finding Alpha

The history of humanity is crowded with seekers of alpha. Financial schemes have been turned into frauds and scams but also engines of growth. Priceless secrets have passed from lip to lip in back alleys and courts. People have bartered, cajoled one another and, when inspired or desperate, invoked the favour of gods. They have sought to be wiser and better than their peers, or to find fruitful niches their peers haven't yet discovered; they've sought to control markets, trade routes and natural resources; they've specialised, advertised, organised, tyrannised; they've laboured, they've contemplated and above all they have innovated.

With all that in mind, what can we say about finding alpha? The first thing we can say is that one should view any guide to finding alpha with a measure of scepticism, at least if one intends to compete in a crowded field.

But keeping Hamming's advice in mind, at a high level, the sources of alpha are information and circumstances, and talent maybe, that allow one to consistently take actions that successfully achieve certain ends. (This notion of alpha applies not only to zero-sum competitions, but to altruistic markets, too.) Bill Benter's end was to pick winning horses (or maybe to make money, or more likely both), and the system he developed (the intricate regression model) and the context in which he deployed it (Hong Kong in the 1990s) allowed him to do that well and consistently.

I can think of a bunch of things that may plausibly help me find alpha (though to be clear I'm not confident in any of these, and even if some correlate with or even cause us to find alpha they may not constitute effective practical advice):

A promising strategy or piece of information, or even a glimpse of one, arrives fused with the fear that someone else will discover it too. That fear was probably one part of Gerald Murnane's mixed feelings at Moonee Valley. But it's a useful fear, because it inspires you to act.

Thanks to Oliver Guest for giving feedback on a draft.

References

Benter, William. 2008. “Computer Based Horse Race Handicapping and Wagering Systems: A Report.” In Efficiency of Racetrack Betting Markets, 183--98. World Scientific.
Bolton, Ruth N, and Randall G Chapman. 1986. “Searching for Positive Returns at the Track: A Multinomial Logit Model for Handicapping Horse Races.” Management Science 32 (8): 1040--60.
Brecher, Steven L. 1980. Beating the Races with a Computer. Software Supply.
Falkenstein, Eric. 2009. Finding Alpha: The Search for Alpha When Risk and Return Break down. Vol. 511. John Wiley & Sons.
Livermore, Jesse L. 1940. How to Trade in Stocks: The Livermore Formula for Combining Time Element and Price. Laurus-Lexecon Kft.
Murnane, Gerald. 2015. Something for the Pain: A Memoir of the Turf. Text Publishing.
Taleb, Nassim Nicholas. 2007. The Black Swan: The Impact of the Highly Improbable. Vol. 2. Random House.

  1. Normally a "handicapper" is someone who assigns horses' handicaps, or maybe sometimes betting odds, but here I take it Benter refers to punters. ↩︎

  2. So you were born Graf Ludwig Wilhelm Franz von Hasenpfeffer and have inherited a vast dynastic fortune, say €1B. You want to invest some of this money, because why not? One thing you can do is invest it all in passively managed index funds: that'll give you a return ~equal to the broader market. But another thing you can do is invest in an active mutual fund. (More likely you'll hire someone to set up a family office in Switzerland and have them take care of all the details, rather than figuring out yourself where to invest and how.) These funds employ professional fund managers who pick investments to try to beat the market (given an acceptable risk level), and in return collect a fee. Alpha is, roughly speaking, a measure of their return minus the return from the passively managed index funds. The actively managed fund needs to produce enough alpha to cover their fees for it to be worth your investing over the passive index funds: it needs to beat the market. ↩︎

  3. Though Murnane never found any alpha betting on horse racing, he certainly found it in fiction writing, being -- as he is -- one of the greatest writers the English language has ever seen. I've previously discussed his writing here and here. ↩︎

  4. Hamming illustrated this with a nice anecdote about Claude Shannon: "Courage is one of the things that Shannon had supremely. You have only to think of his major theorem. He wants to create a method of coding, but he doesn't know what to do so he makes a random code. Then he is stuck. And then he asks the impossible question, 'What would the average random code do?' He then proves that the average code is arbitrarily good, and that therefore there must be at least one good code. Who but a man of infinite courage could have dared to think those thoughts?" ↩︎

  5. Measuring fund performance is a tar pit. Which benchmark do you use? Over which time frame do you measure? How do you estimate costs and fees? How do you address survivorship and backfill bias? How do you adjust for risk? The non-answer is probably "it depends", and in particular I guess it depends on what your goals are. ↩︎

  6. Elsewhere I've read that Livermore declared bankruptcy four times; it doesn't really matter. The point is that that most legendary of traders got wiped out time and again. ↩︎

  7. There were also other reasons why Livermore went bankrupt, including lavish spending, costly divorces and -- so I've read -- failures to adhere to his own system. It's interesting that people (and/or Livermore himself?) have pointed at that latter bit. Hindsight bias is common in gamblers, investors and all other risk-takers. It's always tempting to think, if the trade went well, that the system worked; and if the trade went poorly, that you somehow deviated from the system, meaning its integrity remains inviolate. And maybe this is just a form of motivated reasoning: you do, after all, want your system to work. (People assume that it's easier for them to improve the way they execute their system than it is to improve the system.) That is why it is so important to validate a trading strategy using pre-registered measures, e.g. by walking forward on out-of-sample data, in that way rooting out fragile strategies by bracketing execution and minimising interpretative freedom.

    It is interesting to compare this with scientific research. (Falkenstein 2009): "The thought that I am a sensible, competent person is inconsistent with the thought that I spent most of my creative energy supporting a theory that turned out to be worthless. Therefore, most intellectuals will distort their perception of the data in a tendentious direction, trying not to write off their past efforts as a sunken cost, usually by emphasising not specific results, but the ability of the mathematical framework to accommodate the ultimate true model. [...] The current experts of finance are almost surely smarter, know more math and statistics, and have examined more data, than you have. Yet they also strongly believe in something patently untenable, a strange example of when more expertise leads to a less accurate picture of the world." ↩︎

  8. In reviewing a draft of this post, Oliver Guest pointed out that there are other inefficiencies that matter here, too. Examples include psychological biases disfavouring expected value calculations, and fewer people researching which opportunities are more cost-effective (relative to the number of people researching opportunities in for-profit markets).

    I think those inefficiencies matter, but that selfishness is a larger effect. If everyone were a perfect altruist there'd be lots of money donated to various charities, and lots of effort put into optimizing the way that money is allocated. There'd likely be no 10x or even 5x cash transfer [EA · GW] opportunities, since those would be saturated, because when everyone's a perfect altruist you'd be facing much more competition trying to do good with your resources. (All in all, this would be good for the world, but bad for your prospects of doing lots of good in the world.) ↩︎

  9. Wikipedia writes: "Historian Niall Ferguson agrees that the Rothschilds' couriers did get to London first and alerted the family to Napoleon's defeat, but argues that since the family had been banking on a protracted military campaign, the losses arising from the disruption to their business more than offset any short-term gains in bonds after Waterloo. Rothschild capital did soar, but over a much longer period: Nathan's breakthrough had been prior to Waterloo when he negotiated a deal to supply cash to Wellington's army. The family made huge profits over a number of years from this governmental financing by adopting a high-risk strategy involving exchange-rate transactions, bond-price speculations, and commissions. [...] It is also very commonly reported that the Rothschilds' advanced information was caused by the speed of prized racing pigeons, held by the family. However, this is widely disputed and the Rothschild archive states that, although pigeon post 'was one of the tools of success in the Rothschild business strategy during the period c. 1820--1850, [...] it is likely that a series of couriers on horseback brought the news' of Waterloo to Rothschild."

    This was portrayed by some French socialists as fraud committed by a Jewish capitalist, but a capitalist might say that this is how markets should work: you want the prices of commodities to reflect all available information, and so you must incentivise people to update those prices (by buying or selling the commodities) as soon as new information appears. (Remember, Rothschild did not rely on non-public information; the information was public, and he was just the first to obtain it.) ↩︎

2 comments

Comments sorted by top scores.

comment by romeostevensit · 2023-07-02T02:10:16.819Z · LW(p) · GW(p)

from an old post of mine:

One of the reasons highly useful projects don't get discovered quickly is that they are in under explored spaces. Certain areas are systematically under explored due to biases in peoples' search heuristics. Several examples of such biases are:

  1. Schlep blindness: named by Paul Graham, posits that difficult projects are under explored.

  2. Low-status blindness: projects which are not predicted to bring the project lead prestige are under explored.

  3. High-variance blindness: projects which are unlikely to succeed but that have a positive expected value anyway are under explored.

  4. Already invented blindness: projects that cover areas that have already been explored by others are assumed to have been competently explored.

  5. Not obviously scalable blindness: projects that don't have an obvious route to scaling are under explored.

  6. Over focus on the easy to measure: Inventing new ways of measuring things is often a source of multiple breakthroughs or whole new fields.

comment by Armand_Cognetta · 2023-07-02T04:08:22.865Z · LW(p) · GW(p)

I don’t have anything substantive to say, but I just wanted to comment that I thought this was a wonderful read.