Living in an Inadequate World

post by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2017-11-09T21:23:25.451Z · LW · GW · 48 comments



Follow-up to: Moloch's Toolbox (pt. 1, pt. 2)


Be warned: Trying to put together a background model like the one I sketched in the previous chapter is a pretty perilous undertaking, especially if you don’t have a professional economist checking your work at every stage.

Suppose I offered the following much simpler explanation of how babies are dying inside the US healthcare system:

What if parents don’t really care about their babies?

Maybe parents don’t bond to their babies so swiftly? Maybe they don’t really care that much about those voiceless pink blobs in the early days? Maybe this is one of those things that people think they’re supposed to feel very strongly, and yet the emotion isn’t actually there. Maybe parents just sort of inwardly shrug when their infants die, and only pretend to be sad about it. If they really cared, wouldn’t they demand a system that didn’t kill babies?

In our taxonomy, this would be a “decisionmaker is not beneficiary” explanation, with the parents and doctors being the decisionmakers, and the babies being the beneficiaries.

A much simpler hypothesis, isn’t it?

When we try to do inadequacy analysis, there is such a thing as wrong guesses and false cynicism.

I’m sure there are some parents who don’t bond to their babies all that intensely. I’m sure some of them lie to themselves about that. But in the early days when Omegaven was just plain illegal to sell across state lines, some parents would drive for hours, every month, to buy Omegaven from the Boston Children’s Hospital to take back to their home state. I, for one, would call that an extraordinary effort. Those parents went far outside their routine, beyond what the System would demand of them, beyond what the world was set up to support them doing by default. Most people won’t make an effort that far outside their usual habits even if their own personal lives are at stake.

If parents are letting their babies die of liver damage because the parents don’t care, we should find few extraordinary efforts in these and other cases of baby-saving. This is an observational consequence we can check, and the observational check fails to support the theory.

For a fixed amount of inadequacy, there is only so much dysfunction that needs to be invoked to explain it. By the nature of inadequacy there will usually be more than one thing going wrong at a time… but even so, there’s only a bounded amount of failure to be explained. Every possible dysfunction is competing against every other possible dysfunction to explain the observed data. Sloppy cynicism will usually be wrong, just like your Facebook acquaintances who attribute civilizational dysfunctions to giant malevolent conspiracies.

If you’re sloppy, then you’re almost always going to find some way to conclude, “Oh, those physicists are just part of the broken academic system, what would they really know about the Higgs boson?” You will detect inadequacy every time you go looking for it, whether or not it’s there. If you see the same vision wherever you look, that’s the same as being blind.



In most cases, you won’t need to resort to complicated background analyses to figure out whether something is broken.

I mean, it’s not like the only possible way one might notice that the US health care system is a vast, ill-conceived machine that is broken and also on fire is to understand microeconomics and predict a priori that aspects of this system design might promote inadequate equilibria. In real life, one notices the brokenness by reading economists who blog about the grinding gears and seas of flame, and listening to your friends sob about the screams coming from the ruins.

Then what good does it do to understand Moloch’s toolbox? What’s the point of the skill?

I suspect that for many people, the primary benefit of inadequacy analysis will be in undoing a mistake already made, where they disbelieve in inadequacy even when they’re looking straight at it.

There are people who would simply never try to put up 130 light bulbs in their house—because if that worked, surely some good and diligent professional researcher would have already tried it. The medical system would have made it a standard treatment, right? The doctor would already know about it, right? And sure, sometimes people are stupid, but we’re also people and we’re also stupid so how could we amateurs possibly do better than current researchers on SAD, et cetera.

Often the most commonly applicable benefit from a fancy rational technique will be to cancel out fancy irrationality.1 I expect that the most common benefit of inadequacy analysis will be to break a certain kind of blind trust—that is, trust arrived at by mental reasoning processes that are insensitive to whether you actually inhabit a universe that’s worthy of that trust—and open people’s eyes to the blatant brokenness of things that are easily observed to be broken. Understanding the background theory helps cancel out the elaborate arguments saying that you can’t second-guess the European Central Bank even when it’s straightforward to show how and why they’re making a mistake.

Conversely, I’ve also watched some people plunge straight into problems that I’d guess were inexploitable, without doing the check, and then fail—usually falling prey to the Free Energy Fallacy, supposing that they can win just by doing better on the axis they care about. That subgroup might benefit, not from being told, “Shut up, you’ll always fail, the answer is always no,” but just from a reminder to check for signs of inexploitability.

It may be that some of those people will end up always saying, “I can think of at least one Moloch’s toolbox element in play, therefore this problem will be exploitable!” No humanly possible strictures of rationality can be strict enough to prevent a really determined person from shooting themselves in the foot. But it does help to be aware that the skill exists, before you start refining the skill.

Whether you’re trying to move past modesty or overcome the Free Energy Fallacy:

And then you can move on to step three: the fine-tuning against reality.



In my past experience, I’ve both undershot and overshot the relative competence of doctors in the US medical system:

Anecdote 1: I once became very worried when my then-girlfriend got a headache and started seeing blobs of color, and when she drew the blobs they were left-right asymmetrical. I immediately started worrying about the asymmetry, thinking, “This is the kind of symptom I’d expect if someone had suffered damage to just one side of the brain.” Nobody at the emergency room seemed very concerned, and she waited for a couple of hours to be seen, when I could remember reading that strokes had to be treated within the first few hours (better yet, minutes) to save as much brain tissue as possible.

What she was really experiencing, of course, was her first migraine. And I expect that every nurse we talked to knew that, but only a doctor is allowed to make diagnoses, so they couldn’t legally tell us. I’d read all sorts of wonderful papers about exotic and illuminating forms of brain damage, but no papers about the much more common ailments that people in emergency rooms actually have. “Think horses, not zebras,” as the doctors say.

Anecdote 2: I once saw a dermatologist for a dandruff problem. He diagnosed me with eczema, and gave me some steroid cream to put on my head for when the eczema became especially severe. It didn’t cure the dandruff—but I’d seen a doctor, so I shrugged and concluded that there probably wasn’t much to be done, since I’d already tried and failed using the big guns of the Medical System.

Eight years later, when I was trying to compound a ketogenic meal replacement fluid I’d formulated in an attempt to lose weight, my dandruff seemed to get worse. So I checked whether online paleo blogs had anything to say about treating dandruff via diet. I learned that a lot of dandruff is caused by the Candida fungus (which I’d never heard of), and that the fungus eats ketones. So if switching to a ketogenic diet (or drinking MCT oil, which gets turned into ketones) makes your dandruff worse, why, your dandruff is probably the Candida fungus. I looked up what kills Candida, found that I should use a shampoo containing ketoconazole, kept Googling, found a paper stating that 2% ketocanozole shampoo is an order of magnitude more effective than 1%, learned that only 1% ketocanozole shampoo was sold in the US, and ordered imported 2% Nizoral from Thailand via Amazon. Shortly thereafter, dandruff was no longer a significant issue for me and I could wear dark shirts without constantly checking my right shoulder for white specks. If my dermatologist knew anything about dandruff commonly being caused by a fungus, he never said a word.

From those two data points and others like them, I infer that medical competence—not medical absolute performance, but medical competence relative to what I can figure out by Googling—is high-variance. I shouldn’t trust my doctor on significant questions without checking her diagnosis and treatment plan on the Internet, and I also shouldn’t trust myself.

A lot of the times we put on our inadequacy-detecting goggles, we’re deciding whether to trust some aspect of society to be more competent than ourselves. Part of the point of learning to think in economic terms about this question is to make it more natural to treat it as a technical question where specific lines of evidence can shift specific conclusions to varying degrees.

In particular, you don’t need to be strictly better or worse than some part of society. The question isn’t about ranking people, so you can be smarter in some ways and dumber in others. It can vary from minute to minute as the gods roll their dice.

By contrast, the modest viewpoint seems to me to have a very social-status-colored perspective on such things.

In the modest world, either you think you’re better than doctors and all the civilization backing them, or you admit you’re not as good and that you ought to defer to them.

If you don’t defer to doctors, then you’ll end up as one of those people who try feeding their children organic herbs to combat cancer; the outside view says that that’s what happens to most non-doctors who dare to think they’re better than doctors.

On the modest view, it’s not that we hold up a thumb and eyeball the local competence level, based mostly on observation and a little on economic thinking; and then update on our observed relative performance; and sometimes say, “This varies a lot. I’ll have to check each time.”

Instead, every time you decide whether you think you can do better, you are declaring what sort of person you are.

For an example of what I mean here, consider writer Ozy Brennan’s taxonomy:

I think a formative moment for any rationalist—our “Uncle Ben shot by the mugger” moment, if you will—is the moment you go “holy shit, everyone in the world is fucking insane.” […]

Now, there are basically two ways you can respond to this.

First, you can say “holy shit, everyone in the world is fucking insane. Therefore, if I adopt the radical new policy of not being fucking insane, I can pick up these giant piles of utility everyone is leaving on the ground, and then I win.” […]

This is the strategy of discovering a hot new stock tip, investing all your money, winning big, and retiring to Maui.

Second, you can say “holy shit, everyone in the world is fucking insane. However, none of them seem to realize that they’re insane. By extension, I am probably insane. I should take careful steps to minimize the damage I do.” […]

This is the strategy of discovering a hot new stock tip, realizing that most stock tips are bogus, and not going bankrupt.2

According to this sociological hypothesis, people can react to the discovery that “everyone in the world is insane” by adopting the Maui strategy, or they can react by adopting the not-going-bankrupt strategy.

(Note the inevitable comparison to financial markets—the one part of civilization that worked well enough to prompt an economist, Eugene Fama, to come up with the modern notion of efficiency.)

Brennan goes on to say that these two positions form a “dialectic,” but that nonetheless, some kinds of people are clearly on the “becoming-sane side of things” while others are more on the “insanity-harm-reduction side of things.”

But, speaking first to the basic dichotomy that’s being proposed, the whole point of becoming sane is that your beliefs shouldn’t reflect what sort of person you are. To the extent you’re succeeding, at least, your beliefs should just reflect how the world is.

Good reasoners don’t believe that there are goblins in their closets. The ultimate reason for this isn’t that goblin-belief is archaic, outmoded, associated with people lost in fantasy worlds, too much like wishful thinking, et cetera. It’s just that we opened up our closets and looked and we didn’t see any goblins.

The goal is simply to be the sort of person who, in worlds with closet goblins, ends up believing in closet goblins, and in worlds without closet goblins, ends up disbelieving in closet goblins. Avoiding beliefs that sound archaic does relatively little to help you learn that there are goblins in a world where goblins exist, so it does relatively little to establish that there aren’t goblins in a world where they don’t exist. Examining particular empirical predictions of the goblin hypothesis, on the other hand, does provide strong evidence about what world you’re in.

To reckon with the discovery that the world is mad, Brennan suggests that we consider the mix of humble and audacious “impulses in our soul” and try to strike the right balance. Perhaps we have some personality traits or biases that dispose us toward believing in goblins, and others that dispose us toward doubting them. On this framing, the heart of the issue is how we can resolve this inner conflict; the heart isn’t any question about the behavioral tendencies or physiology of goblins.

This is a central disagreement I have with modest epistemology: modest people end up believing that they live in an inexploitable world because they’re trying to avoid acting like an arrogant kind of person. Under modest epistemology, you’re not supposed to adapt rapidly and without hesitation to the realities of the situation as you observe them, because that would mean trusting yourself to assess adequacy levels; but you can’t trust yourself, because Dunning-Kruger, et cetera.

The alternative to modest epistemology isn’t an immodest epistemology where you decide that you’re higher status than doctors after all and conclude that you can now invent your own de novo medical treatments as a matter of course. The alternative is deciding for yourself whether to trust yourself more than a particular facet of your civilization at this particular time and place, checking the results whenever you can, and building up skill.

When it comes to medicine, I try to keep in mind that anyone whatsoever with more real-world medical experience may have me beat cold solid when it comes to any real-world problem. And then I go right on double-checking online to see if I believe what the doctor tells me about whether consuming too much medium-chain triglyceride oil could stress my liver.3

In my experience, people who don’t viscerally understand Moloch’s toolbox and the ubiquitously broken Nash equilibria of real life and how group insanity can arise from intelligent individuals responding to their own incentives tend to unconsciously translate all assertions about relative system competence into assertions about relative status. If you don’t see systemic competence as rare, or don’t see real-world systemic competence as driven by rare instances of correctly aligned incentives, all that’s left is status. All good and bad output is just driven by good and bad individual people, and to suggest that you’ll have better output is to assert that you’re individually smarter than everyone else. (This is what status hierarchy feels like from the inside: to perform better is to be better.)

On a trip a couple of years ago to talk with the European existential risk community, which has internalized norms from modest epistemology to an even greater extent than the Bay Area community has, I ran into various people who asked questions like, “Why do you and your co-workers at MIRI think you can do better than academia?” (MIRI is the Machine Intelligence Research Institute, the organization I work at.)

I responded that we were a small research institute that sustains itself on individual donors, thereby sidestepping a set of standard organizational demands that collectively create bad incentives for the kind of research we’re working on. I described how we had deliberately organized ourselves to steer clear of incentives that discourage long-term substantive research projects, to avoid academia’s “publish or perish” dynamic, and more generally to navigate around the multiple frontiers of competitiveness where researchers have to spend all their energy competing along those dimensions to get into the best journals.

These are known failure modes that academics routinely complain about, so I wasn’t saying anything novel or clever. The point I wanted to emphasize was that it’s not enough to say that you want risky long-term research in the abstract; you have to accept that your people won’t be at the competitive frontier for journal publications anymore.

The response I got back was something like a divide-by-zero error. Whenever I said “the nonprofit I work at has different incentives that look prima facie helpful for solving this set of technical problems,” my claim appeared to get parsed as “the nonprofit I work at is better (higher status, more authoritative, etc.) than academia.”

I think that the people I was talking with had already internalized the mathematical concept of Nash equilibria, but I don’t think they were steeped in a no-free-energy microeconomic equilibrium view of all of society where most of the time systems end up dumber than the people in them due to multiple layers of terrible incentives, and that this is normal and not at all a surprising state of affairs to suggest. And if you haven’t practiced thinking about organizations’ comparative advantages from that perspective long enough to make that lens more cognitively available than the status comparisons lens, then it makes sense that all talk of relative performance levels between you and doctors, or you and academia, or whatever, will be autoparsed by the easier, more native, more automatic status lens.

Because, come on, do you really think you’re more authoritative / respectable / qualified / reputable / adept than your doctor about medicine? If you think that, won’t you start consuming Vitamin C megadoses to treat cancer? And if you’re not more authoritative / respectable / qualified / reputable / adept than your doctor, then how could you possibly do better by doing Internet research?

(Among most people I know, the relative status feeling frequently gets verbalized in English as “smarter,” so if the above paragraph didn’t make sense, try replacing the social-status placeholder “authoritative / respectable / etc.” with “smarter.”)

Again, a lot of the benefit of becoming fluent with this viewpoint is just in having a way of seeing “systems with not-all-that-great outputs,” often observed extensively and directly, that can parse into something that isn’t “Am I higher-status (‘smarter,’ ‘better,’ etc.) than the people in the system?”



I once encountered a case of (honest) misunderstanding from someone who thought that when I cited something as an example of civilizational inadequacy (or as I put it at the time, “People are crazy and the world is mad”), the thing I was trying to argue was that the Great Stagnation was just due to unimpressive / unqualified / low-status (“stupid”) scientists.4 He thought I thought that all we needed to do was take people in our social circle and have them go into biotech, or put scientists through a CFAR unit, and we’d see huge breakthroughs.5

What?” I said.

(I was quite surprised.)

“I never said anything like that,” I said, after recovering from the shock. “You can’t lift a ten-pound weight with one pound of force!”

I went on to say that it’s conceivable you could get faster-than-current results if CFAR’s annual budget grew 20x, and then they spent four years iterating experimentally on techniques, and then a group of promising biotechnology grad students went through a year of CFAR training…6

So another way of thinking about the central question of civilizational inadequacy is that we’re trying to assess the quantity of effort required to achieve a given level of outperformance. Not “Can it be done?” but “How much work?”

This brings me to the single most obvious notion that correct contrarians grasp, and that people who have vastly overestimated their own competence don’t realize: It takes far less work to identify the correct expert in a pre-existing dispute between experts, than to make an original contribution to any field that is remotely healthy.

I did not work out myself what would be a better policy for the Bank of Japan. I believed the arguments of Scott Sumner, who is not literally mainstream (yet), but whose position is shared by many other economists. I sided with a particular band of contrarian expert economists, based on my attempt to parse the object-level arguments, observing from the sidelines for a while to see who was right about near-term predictions and picking up on what previous experience suggested were strong cues of correct contrarianism.7

And so I ended up thinking that I knew better than the Bank of Japan. On the modest view, that’s just about as immodest as thinking you can personally advance the state of the art, since who says I ought to be smarter than the Bank of Japan at picking good experts to trust, et cetera?

But in real life, inside a civilization that is often tremendously broken on a systemic level, finding a contrarian expert seeming to shine against an untrustworthy background is nowhere remotely near as difficult as becoming that expert yourself. It’s the difference between picking which of four runners is most likely to win a fifty-kilometer race, and winning a fifty-kilometer race yourself.

Distinguishing a correct contrarian isn’t easy in absolute terms. You are still trying to be better than the mainstream in deciding who to trust.8 For many people, yes, an attempt to identify contrarian experts ends with them trusting faith healers over traditional medicine. But it’s still in the range of things that amateurs can do with a reasonable effort, if they’ve picked up on unusually good epistemology from one source or another.

We live in a sufficiently poorly-functioning world that there are many visibly correct contrarians whose ideas are not yet being implemented in the mainstream, where the authorities who allegedly judge between experts are making errors that appear to me trivial. (And again, by “errors,” I mean that these authorities are endorsing factually wrong answers or dominated policies—not that they’re passing up easy rewards given their incentives.)

In a world like that, you can often know things that the average authority doesn’t know… but not because you figured it out yourself, in almost every case.



Going beyond picking the right horse in the race and becoming a horse yourself, inventing your own new personal solution to a civilizational problem, requires a much greater investment of effort.

I did make up my own decision theory—not from a tabula rasa, but still to my own recipe. But events like that should be rare in a given person’s life. Logical counterfactuals in decision theory are one of my few major contributions to an existing academic field, and my early thoughts on this topic were quickly improved on by others.9 And that was a significant life event, not the sort of thing I believe I’ve done every month.

Above all, reaching the true frontier requires picking your battles.

Computer security professionals don’t attack systems by picking one particular function and saying, “Now I shall find a way to exploit these exact 20 lines of code!” Most lines of code in a system don’t provide exploits no matter how hard you look at them. In a large enough system, there are rare lines of code that are exceptions to this general rule, and sometimes you can be the first to find them. But if we think about a random section of code, the base rate of exploitability is extremely low—except in really, really bad code that nobody looked at from a security standpoint in the first place.

Thinking that you’ve searched a large system and found one new exploit is one thing. Thinking that you can exploit arbitrary lines of code is quite another.

No matter how broken academia is, no one can improve on arbitrary parts of the modern academic edifice. My own base frequency for seeing scholarship that I think I can improve upon is “almost never,” outside of some academic subfields dealing with the equivalent of “unusually bad code.” But don’t expect bad code to be guarding vaults of gleaming gold in a form that other people value, except with a very low base rate. There do tend to be real locks on the energy-containing vaults not already emptied… almost (but not quite) all of the time.

Similarly, you do not generate a good startup idea by taking some random activity, and then talking yourself into believing you can do it better than existing companies. Even where the current way of doing things seems bad, and even when you really do know a better way, 99 times out of 100 you will not be able to make money by knowing better. If somebody else makes money on a solution to that particular problem, they’ll do it using rare resources or skills that you don’t have—including the skill of being super-charismatic and getting tons of venture capital to do it.

To believe you have a good startup idea is to say, “Unlike the typical 99 cases, in this particular anomalous and unusual case, I think I can make a profit by knowing a better way.”

The anomaly doesn’t have to be some super-unusual skill possessed by you alone in all the world. That would be a question that always returned “No,” a blind set of goggles. Having an unusually good idea might work well enough to be worth trying, if you think you can standardly solve the other standard startup problems. I’m merely emphasizing that to find a rare startup idea that is exploitable in dollars, you will have to scan and keep scanning, not pursue the first “X is broken and maybe I can fix it!” thought that pops into your head.

To win, choose winnable battles; await the rare anomalous case of, “Oh wait, that could work.”



In 2014, I experimentally put together my own ketogenic meal replacement drink via several weeks of research, plus months of empirical tweaking, to see if it could help me with long-term weight normalization.

In that case, I did not get to pick my battleground.

And yet even so, I still tried to design my own recipe. Why? It seems I must have thought I could do better than the best ketogenic liquid-food recipes that had ever before been tried, as of 2014. Why would I believe I could do the best of anyone who’s yet tried, when I couldn’t pick my battle?

Well, because I looked up previous ketogenic Soylent recipes, and they used standard multivitamin powders containing, e.g., way too much manganese and the wrong form of selenium. (You get all the manganese you need from ordinary drinking water, if it hasn’t been distilled or bottled. Excess amounts may be neurotoxic. One of the leading hypotheses for why multivitamins aren’t found to produce net health improvement, despite having many individual components found to be helpful, is that multivitamins contain 100% of the US RDA of manganese. Similarly, if a multivitamin includes sodium selenite instead of, e.g., se-methyl-selenocysteine, it’s the equivalent of handing you a lump of charcoal and saying, “You’re a carbon-based lifeform; this has carbon in it, right?”)

Just for the sake of grim amusement, I also looked up my civilization’s medically standard ketogenic dietary options—e.g., for epileptic children. As expected, they were far worse than the amateur Soylent-inspired recipes. They didn’t even contain medium-chain triglycerides, which your liver turns directly into ketones. (MCT is academically recommended, though not commercially standard, as the basis for maintaining ketosis in epileptic children.) Instead the retail dietary options for epileptic children involved mostly soybean oil, of which it has been said, “Why not just shoot them?”

Even when we can’t pick our battleground, sometimes the most advanced weapon on offer turns out to be a broken stick and it’s worth the time to carve a handaxe.

… But even then, I didn’t try to synthesize my own dietary theory from scratch. There is nothing I believe about how human metabolism works that’s unique or original to me. Not a single element of my homemade Ketosoylent was based on my personal, private theory of how any of the micronutrients worked. Who am I to think I understand Vitamin D3 better than everyone else in the world?

The Ketosoylent didn’t work for long-term weight normalization, alas—the same result as all other replicated experiments on trying to long-term-normalize weight via putting different things inside your mouth. (The Shangri-La Diet I mentioned at the start of this book didn’t work for me either.)

So it goes. I mention the Ketosoylent because it’s the most complicated thing I’ve tried to do without tons of experience in a domain and without being able to pick my battles.

In the simpler and happier case of treating Brienne’s Seasonal Affective Disorder, I again didn’t get to pick the battleground; but SAD has received far less scientific attention to date than obesity. And success there again didn’t involve coming up with an amazing new model of SAD. It’s not weird and private knowledge that sufficiently bright light might cure SAD. The Sun is known to work almost all the time.

So a realistic lifetime of trying to adapt yourself to a broken civilization looks like:

The accumulation of many judgments of the latter kind is where you get the fuel for many small day-to-day decisions (e.g., about what to eat), and much of your ability to do larger things (like solving a medical problem after going through the medical system has proved fruitless, or executing well on a startup).



A few final pieces of advice on everyday thinking about inadequacy:

When it comes to estimating the competence of some aspect of civilization, especially relative to your own competence, try to update hard on your experiences of failure and success. One data point is a hell of a lot better than zero data points.

Worrying about how one data point is “just an anecdote” can make sense if you’ve already collected thirty data points. On the other hand, when you previously just had a lot of prior reasoning, or you were previously trying to generalize from other people’s not-quite-similar experiences, and then you collide directly with reality for the first time, one data point is huge.

If you do accidentally update too far, you can always re-update later when you have more data points. So update hard on each occasion, and take care not to flush any new observation down the toilet.

Oh, and bet. Bet on everything. Bet real money. It helps a lot with learning.

I once bet $25 at even odds against the eventual discovery of the Higgs boson—after 90% of the possible mass range had been experimentally eliminated, because I had the impression from reading diatribes against string theory that modern theoretical physics might not be solid enough to predict a qualitatively new kind of particle with prior odds greater than 9:1.

When the Higgs boson was discovered inside the remaining 10% interval of possible energies, I said, “Gosh, I guess they can predict that sort of thing with prior probability greater than 90%,” updated strongly in favor of the credibility of things like dark matter and dark energy, and then didn’t make any more bets like that.

I made a mistake; and I bet on it. This let me experience the mistake in a way that helped me better learn from it. When you’re thinking about large, messy phenomena like “the adequacy of human civilization at understanding nutrition,” it’s easy to get caught up in plausible-sounding stories and never quite get around to running the experiment. Run experiments; place bets; say oops. Anything less is an act of self-sabotage.



Next: Blind Empiricism.

The full book will be available November 16th. You can go to to pre-order the book, or sign up for notifications about new chapters and other developments.



  1. As an example, relatively few people in the world need well-developed skills at cognitive reductionism for the purpose of disassembling aspects of nature. The reason why anyone else needs to learn cognitive reductionism—the reason it’s this big public epistemic hygiene issue—is that there are a lot of damaging supernatural beliefs that cognitive reductionism helps counter. 

  2. Brennan, “The World Is Mad.”

    When I ran a draft of this chapter by Brennan, they said that they basically agree with what I’m saying here, but are thinking about these issues using a different conceptual framework. 

  3. Answer: this is the opposite of standard theory; she was probably confusing MCT with other forms of saturated fat. 

  4. The Great Stagnation is economist Tyler Cowen’s hypothesis that declining rates of innovation since the 1970s (excluding information technology, for the most part) have resulted in relative economic stagnation in the developed world. 

  5. CFAR, the Center for Applied Rationality, is a nonprofit that applies ideas from cognitive science to everyday problem-solving and decision-making, running workshops for people who want to get better at solving big global problems. MIRI and CFAR are frequent collaborators, and share office space; the organization’s original concept came from MIRI’s work on rationality. 

  6. See also Weinersmith’s Law: “No problem is too hard. Many problems are too fast.” 

  7. E.g., the cry of “Stop ignoring your own carefully gathered experimental evidence, damn it!” 

  8. Though, to be clear, the mainstream isn’t actually deciding who to trust. It’s picking winners by some other criterion that on a good day is not totally uncorrelated with trustworthiness. 

  9. In particular, Wei Dai came up with updatelessness, yielding the earliest version of what's now called functional decision theory. See Soares and Levinstein's “Cheating Death in Damascus” for a description. 


Comments sorted by top scores.

comment by Kaj_Sotala · 2017-11-10T13:57:35.794Z · LW(p) · GW(p)
I once encountered a case of (honest) misunderstanding from someone who thought that when I cited something as an example of civilizational inadequacy (or as I put it at the time, “People are crazy and the world is mad”), the thing I was trying to argue was that the Great Stagnation was just due to unimpressive / unqualified / low-status (“stupid”) scientists. He thought I thought that all we needed to do was take people in our social circle and have them go into biotech, or put scientists through a CFAR unit, and we’d see huge breakthroughs.

Datapoint: I also totally thought that by "people are crazy and the world is mad", you meant something like this too... in fact, it wasn't until this sequence that I became convinced for certain that you didn't mean that. E.g. a lot of the Eld Science stuff in the Old Sequences seemed to be saying something like "the rules of science aren't strict enough and if scientists just cared enough to actually make an effort and try to solve the problem, rather than being happy to meet the low bar of what's socially demanded of them, then science would progress a lot faster". A bunch of stuff in HPMOR seemed to have this vibe, too; giving the strong impression that most societal problems are due to failures of individual rationality, and which could be fixed if people just cared enough.

Replies from: Benito, sarahconstantin, Gavin
comment by Ben Pace (Benito) · 2017-11-10T17:36:29.574Z · LW(p) · GW(p)

Yes, I explicitly remember Jeffryssai's answer to the question of what Einstein & co. got wrong was that "They thought it was acceptable to take 50 years to develop the next big revolution" or something, and I recall my takeaway from that being "If you think it's okay to doss around and just think about fun parts of physics, as opposed to trying to figure out the critical path to the next big insight and working hard on that, then you'll fail to save humanity". And the 'if people just cared enough' mistake is also one I held for a long time - it wasn't until I met it's strong form in EA that I realised that the problem isn't that people don't care enough.

comment by sarahconstantin · 2017-11-13T06:14:03.855Z · LW(p) · GW(p)

Interestingly enough, James Watson (as in Watson & Crick) does literally think that the problem with biology is that biologists today don't work hard enough, that they just don't spend enough hours a week in the lab.

I'm not sure this is true. But even if it were true, you could still view it as an incentive problem rather than a character problem. (The fact that there's enough money & prestige in science for it to serve as an even remotely plausible career to go into for reasons of vanilla personal advancement means that it'll attract some people who are vanilla upper-middle-class strivers rather than obsessive truth-seekers, and those people will work less intensely.) You wouldn't fix the problem by injecting ten workaholics into the pool of researchers.

comment by Gavin · 2017-11-10T22:08:11.533Z · LW(p) · GW(p)

Isn't this true in a somewhat weaker form? It takes individuals and groups putting in effort at personal risk to move society forward. The fact that we are stuck in inadequate equilibriums is evidence that we have not progressed as far as we could.

Scientists moving from Elsevier to open access happened because enough of them cared enough to put in the effort and take the risk to their personal success. If they had cared a little bit more on average, it would have happened earlier. If they had cared a little less, maybe it would have taken a few more years.

If humans had 10% more instinct for altruism, how many more of these coordination problems would alreadybe solved? There is a deficit of caring about solving civilizational problems. That doesn't change the observation that most people are reacting to their own incentives and we can't really blame them.

Replies from: RobbBB
comment by Rob Bensinger (RobbBB) · 2017-11-10T22:43:47.611Z · LW(p) · GW(p)

Yeah, this isn't obviously wrong from where I'm standing:

"the rules of science aren't strict enough and if scientists just cared enough to actually make an effort and try to solve the problem, rather than being happy to meet the low bar of what's socially demanded of them, then science would progress a lot faster"

But it's imprecise. Eliezer is saying that the amount of extra individual effort, rationality, creative institution redesign, etc. to yield significant outperformance isn't trivial. (In my own experience, people tend to put too few things in the "doable but fairly difficult" bin, and too many things in the "fairly easy" and "effectively impossible" bins.)

Eliezer is also saying that the dimension along which you're trying to improve science makes a huge difference. E.g., fields like decision theory may be highly exploitable in AI-grade solutions and ideas even if biomedical research turns out to be more or less inexploitable in cancer cures. (Though see Sarah Constantin's "Is Cancer Progress Stagnating?" and follow-up posts on that particular example.)

Replies from: ialdabaoth
comment by ialdabaoth · 2017-11-11T00:33:18.653Z · LW(p) · GW(p)

If you want to find hidden inefficiencies to exploit, don't look for unknown slopes; look for pixelation along the boundaries of well-worn maps.

comment by namespace (ingres) · 2017-11-10T06:36:44.319Z · LW(p) · GW(p)
I looked up what kills Candida, found that I should use a shampoo containing ketoconazole, kept Googling, found a paper stating that 2% ketocanozole shampoo is an order of magnitude more effective than 1%, learned that only 1% ketocanozole shampoo was sold in the US, and ordered imported 2% Nizoral from Thailand via Amazon. Shortly thereafter, dandruff was no longer a significant issue for me and I could wear dark shirts without constantly checking my right shoulder for white specks. If my dermatologist knew anything about dandruff commonly being caused by a fungus, he never said a word.

So I don't want to seem like I'm missing the point here, because I do understand that it's about the high variance of medical care, however:

I had this same issue, and did not have to import my shampoo from Thailand. I went to the doctor complaining about incredible itching and dandruff, the doctor ran her hand through my hair and diagnosed a severe fungal infection. They then wrote a magic order for 2% Nizoral and I picked it up at the pharmacy. Saying it's "not sold in the US" makes it sound like there's some sort of total ban on the thing, as opposed to it being prescription only. And while I get that if you don't have a doctor to help you then prescription-only may as well be "totally banned", they're still not the same thing from the perspective of someone who might one day catch horrible hair fungus.

The basic point I'm trying to make is that the system isn't quite as broken as I feel like you're portraying it here. If for some reason you disbelieve me, I have the prescription and the shampoo bottle.

Replies from: RobbBB
comment by Rob Bensinger (RobbBB) · 2017-11-10T06:52:12.427Z · LW(p) · GW(p)
The basic point I'm trying to make is that the system isn't quite as broken as I feel like you're portraying it here.

Do you know why Nizoral is prescription-only in the US? The clarification you made is useful, but it sounds like you're treating "Nizoral is prescription-only" as indicative of a less broken system than just "Nizoral happens to not be available in the US," where I have the opposite intuition. Is Nizoral really dangerous?

Replies from: ChristianKl, ingres
comment by ChristianKl · 2017-11-10T17:22:56.367Z · LW(p) · GW(p)

If you want to treat the illness dosing the stuff higher is likely helping you. Having it at 5% is likely even more successful than having it at 2%. On the other hand, it also increases side-effects. If it wouldn't The stuff isn't good for the liver, so there a reason to discourage people from overdosing it.

Somewhere there's likely a Nizoral dose that is actually dangerous. The general philosophy that there should be a dose that has to be expert approved isn't that bad. I would prefer a more open system but it's intellectually defensible.

comment by namespace (ingres) · 2017-11-10T07:04:58.462Z · LW(p) · GW(p)

>"Nizoral happens to not be available in the US,"

That's...not a real state of the world. If something that common weren't somehow explicitly regulated I'm sure it would be available OTC. As a consequence, given not available my prediction options are between "so regulated that it's literally just not a thing in the US" and "regulated enough that you can't just buy it off Amazon, but still available by doctor".

As for danger, from what I read Nizoral is not going to be a good time for your liver if you ate it somehow.

comment by waveman · 2017-11-09T22:14:03.329Z · LW(p) · GW(p)

For "decision-maker is not beneficiary" a possibly better term is Nassim Taleb's "No Skin in the Game".

comment by Ben Pace (Benito) · 2017-11-12T00:09:17.831Z · LW(p) · GW(p)

I'm promoting this to Featured, because of the two key insights I got from this. The first was the crystallisation of your reply to modest epistemology - neither a rejection of 'experts' nor putting full trust in them, but instead building skills in modelling institutions, markets, incentives and adequacy, in the specific domains you care about. And the second was the sections on the distinction between being able to pick the best experts, and being able to do better than the best experts, and the rarity of the latter relative to the former.

comment by Paul Crowley (ciphergoth) · 2017-11-10T06:04:09.026Z · LW(p) · GW(p)

Looks like this hasn't been marked as part of the "INADEQUATE EQUILIBRIA" sequence: unlike the others, it doesn't carry this banner, and it isn't listed in the TOC.

Replies from: habryka4
comment by habryka (habryka4) · 2017-11-12T21:17:24.263Z · LW(p) · GW(p)

Fixed now. Sorry for that!

comment by jsalvatier · 2017-11-10T08:39:23.185Z · LW(p) · GW(p)

This is great, I love it

comment by NancyLebovitz · 2017-11-11T12:08:06.985Z · LW(p) · GW(p)

I think the long history of "getting the homeless ready for housing" rather than just giving them housing is an example of civilizational inadequacy.

comment by skybrian · 2017-11-11T04:14:32.821Z · LW(p) · GW(p)

I'd like to see citations for the claims about maganese and selenium.

Replies from: waveman
comment by waveman · 2017-11-11T11:21:23.073Z · LW(p) · GW(p)

A bit of poking around later ... (because I had been taking some Manganese to help with some bone problems. The recomended intake is maybe 1.5mg/day (adult male, more for pregnant women)). The toxic level is around 11mg/day. Among other things it can be neurotoxic. So it has pretty tight margins, similar to, say Fluorine. See links from the wikipedia article.

>(EY) You get all the manganese you need from ordinary drinking water, if it hasn’t been distilled or bottled.

This really depends. Amounts in drinking water vary wildly from hardly any to far too much. Brown staining on porcelain is one sign that your water is over 0.5mg/Liter, and the taste becomes 'undesirable' at that point too. See

and "Intellectual Impairment in School-Age Children Exposed to Manganese from Drinking Water" on (LW mangles the URL sorry)

Groundwater or runoff water affected by pollution seems to make it much more likely Mn is too high.

comment by itaibn0 · 2017-11-12T23:20:34.888Z · LW(p) · GW(p)

(Background: I used to be skeptical about AI risk as a high-value cause, now I am uncertain, and I am still skeptical of MIRI.)

I disagree with you about MIRI compared with mainstream academia. Academics may complain about the way academia discourages "long-term substantive research projects", but taking a broader perspective academia is still the best thing there is for such projects. I think you misconstrued comments by academics complaining about their situation on the margin as being statements about academia in the absolute, and thereby got the wrong idea about the relative difficulty of doing good research within and outside academia.

When you compete for grant funding, that means your work is judged by people with roughly the same level of expertise as you. When you make a publically-funded research institute your work is judged for more shallowly. That you chose to go along the second path rather than the first path had left a bad first impression on me when I first learned of it, like you can't make a convincing case in a fair test. As MIRI grew and as I learned more about it, I got the impression that since MIRI is a small team too little contact with a broader intellectual community it prematurely reached a consensus on a particular set of approaches and assumptions that I think are likely to go nowhere.

Replies from: RobbBB
comment by Rob Bensinger (RobbBB) · 2017-11-13T03:25:23.996Z · LW(p) · GW(p)

I think we agree that relying on funding sources with a shallower understanding of your output is worse, all else being equal, and that it can create bad incentives to optimize for shallow signs of quality.

We might disagree about how shallow MIRI's actual evaluations have ended up being, or about how much depth similar groups like FHI bought by working within academia.

We might also disagree about the total amount of attention that should be going to these issues. Robin Hanson is one of the world's leading researchers in this space advocating a decidedly non-MIRI perspective, and according to a blog post he wrote last year, prior to the Open Philanthropy Project (in 2016) he had received no research support for any of his futurism-related work in his entire career. FHI exists, so there wasn't literally zero funding available for serious work on AI outcomes within academia over the past 20+ years; but I don't think the empirical adequacy of academia at funding the best work in this class is high enough to indicate that there should have been zero money spent on any future-of-AI work outside of academia between 2000 and 2015.

I got the impression that since MIRI is a small team too little contact with a broader intellectual community it prematurely reached a consensus on a particular set of approaches and assumptions that I think are likely to go nowhere.

When I think about the people working on AGI outcomes within academia these days, I think of people like Robin Hanson, Nick Bostrom, Stuart Russell, and Eric Drexler, and it's not immediately obvious to me that these people have converged more with each other than any of them have with researchers at MIRI. Maybe it would help here if you gave examples of ways you think MIRI's on the wrong track, since it might be that your central objection is to the dimensions along which MIRI folks tend to disagree with other groups, rather than to the magnitude of disagreement on its own.

Replies from: itaibn0, itaibn0, itaibn0
comment by itaibn0 · 2017-11-14T12:22:41.759Z · LW(p) · GW(p)

Some meta-level comments and questions:

This discussion has moved far off-topic away from EY's general rationality lessons. I'm pleased with this, since these are topics that I want to discuss, but I want to mention this explicitly since constant topic-changes can be bad for a productive discussion by preventing the participants from going into any depth. In addition, lurkers might be annoyed at reading yet another AI argument. Do you think we should move the discussion to a different venue?

My motivations for discussing this are a chance to talk about critisms of MIRI that I haven't gotten down in writing in detail before, a chance to get a rough impression on how MIRI supporters to these explanations, and more generally an opportunity to practice intellectual honest debates. I don't expect the discussion to go on far enough to resolve our disagreements, but I am trying to anyways to get practice. I'm currently enthusiastic about continuing the discussion. but the sort of enthusiasm that could easily wane in a day. What is your motivation?

comment by itaibn0 · 2017-11-13T14:44:37.512Z · LW(p) · GW(p)

Thinking further, I've spotted something that may a crucial misunderstanding. Is the issue whether EY was right to create his own technical research institute on AI risk, is it whether he was right to pursue AI risk at all? I agree that before EY there was relatively little academic work on AI risk, and that he played an important role in increasing the amount of attention the issue recieves. I think it would have been a mistake for him to ignore the issue on the basis that the experts must know better than him and they aren't worried.

On the other hand, I expect an equally well-funded and well-staffed group that is mostly within academia to do a better job than MIRI. I think EY was wrong in believing that he could create an institute that is better at pursuing long-term technical research in a particular topic than academia.

comment by itaibn0 · 2017-11-13T13:35:59.802Z · LW(p) · GW(p)
When I think about the people working on AGI outcomes within academia these days, I think of people like Robin Hanson, Nick Bostrom, Stuart Russell, and Eric Drexler, and it's not immediately obvious to me that these people have converged more with each other than any of them have with researchers at MIRI.

I see the lack of convergence between people in academia as supporting my position, since I am claiming that MIRI is looking too narrowly. I think AI risk research is still in a brainstorming stage where we still don't have a good grasp on what all the possibilities are. If all of these people have rather different ideas for how to go about it, was is it just the approaches that Eliezer Yudkowsky likes that are getting all the funding?

I also have specific objections. Let's take TDT and FDT as an example since they were mentioned in the post. The primary motivation for them is that they handle Newcombe-like dilemmas better. I don't think Newcombe-like dilemmas are relevant for the reasoning of potentially dangerous AIs, and I don't think you will get a good holistic understanding of what a good reasoner out of these theories. One secondary motivation for TDT/UDT/FDT is a fallacious argument that it endorses cooperation in the true prisoner's dilemma. Informal arguments seem to be the load-bearing applying these theories to any particular problem; the technical works seem to be mainly formalizing narrow instances of these theories to agree with the informal intuition. I don't know about FDT, but a fundamental assumption behind TDT and UDT is the existence of a causal structure behind logical statements, which sounds implausible to me.

Replies from: RobbBB, RobbBB
comment by Rob Bensinger (RobbBB) · 2017-11-13T19:04:07.282Z · LW(p) · GW(p)
I don't think Newcombe-like dilemmas are relevant for the reasoning of potentially dangerous AIs

When a programmer writes software, it's because they have a prediction in mind about how the software is likely to behave in the future: we have goals we want software to achieve, and we write the code that we think will behave in the intended way. AGI systems are particularly likely to end up in Newcomblike scenarios if we build them to learn our values by reasoning about their programmers' intentions and goals, or if the system constructs any intelligent subprocesses or subagents to execute tasks, or executes significant self-modifications at all. In the latter cases, the system itself is then in a position of designing reasoning algorithms based on predictions about how the algorithms will behave in the future.

The same principle holds if two agents are modeling each other in real time, as opposed to predicting a future agent; e.g., two copies of an AGI system, or subsystems of a single AGI system. The copies don't have to be exact, and the systems don't have to have direct access to each other's source code, for the same issues to crop up.

One secondary motivation for TDT/UDT/FDT is a fallacious argument that it endorses cooperation in the true prisoner's dilemma.

What's the fallacy you're claiming?

Informal arguments seem to be the load-bearing applying these theories to any particular problem; the technical works seem to be mainly formalizing narrow instances of these theories to agree with the informal intuition.

This seems wrong, if you're saying that we can't formally establish the behavior of different decision theories, or that applying theories to different cases requires ad-hoc emendations; see section 5 of "Functional Decision Theory" (and subsequent sections) for a comparison and step-by-step walkthrough of procedures for FDT, CDT, and EDT. One of the advantages we claim for FDT over CDT and EDT is that it doesn't require ad-hoc tailoring for different dilemmas (e.g., ad-hoc precommitment methods or ratification procedures, or modifications to the agent's prior).

I don't know about FDT, but a fundamental assumption behind TDT and UDT

"UDT" is ambiguous and has been used to refer to a lot of different things, but Wei Dai's original proposals of UDT are particular instances of FDT. FDT can be thought of as a generalization of Wei Dai's first versions of UDT, that makes fewer commitments than Wei Dai's particular approach.

but a fundamental assumption behind TDT and UDT is the existence of a causal structure behind logical statements, which sounds implausible to me.

None of the theories mentioned make any assumption like that; see the FDT paper above.

Replies from: itaibn0, itaibn0
comment by itaibn0 · 2017-11-14T14:02:58.119Z · LW(p) · GW(p)
What's the fallacy you're claiming?

First, to be clear, I am referring to things such as this description of the prisoner's dilemma and EY's claim that TDT endorses cooperation. The published material has been careful to only say that these decision theories endorse cooperation among identical copies running the same source code, but as far as I can tell some researchers at MIRI still believe this stronger claim and this claim has been a major part of the public perception of these decision theories (example here; see section II).

The problem is that when two FDT agent with a different utility functions and different prior knowledge are facing a prisoner's dilemma with each other, then their decisions are actually two different logical variables X0 and X1. The argument for cooperating is that X0 and X1 are sufficiently similar to one another that in the counterfactual where X0=C we also have X1=C. However, you could just as easily take the opposite premise, where X0 and X1 are sufficiently dissimilar that counterfactually changing X0 will have no effect on X1. Then you are left with the usual CDT analysis of the game. Given the vagueness of logical counterfactuals it is impossible to distinguish these two situations.

Here's a related question: What does FDT say about the centipede game? There's no symmetry between the players so I can't just plug in the formalism. I don't see how you can give an answer that's in the spirit of cooperating in the prisoner's dilemma without reaching the conclusion that FDT involves altruism among all FDT agents through some kind of veil of ignorance argument. And taking that conclusion is counter to the affine-transformation-invariance of utility functions.

comment by itaibn0 · 2017-11-14T00:33:13.273Z · LW(p) · GW(p)
"but a fundamental assumption behind TDT and UDT is the existence of a causal structure behind logical statements, which sounds implausible to me."
None of the theories mentioned make any assumption like that; see the FDT paper above.

Page 14 of the FDT paper:

Instead of a do operator, FDT needs a true operator, which takes a logical sentence φ and updates P to represent the scenario where φ is true...
...Equation (4) works given a graph that accurately describes how changing the value of a logical variable affects other variables, but it is not yet clear how to construct such a thing—nor even whether it can be done in a satisfactory manner within Pearl’s framework.

This seems wrong, if you're saying that we can't formally establish the behavior of different decision theories, or that applying theories to different cases requires ad-hoc emendations; see section 5 of "Functional Decision Theory" (and subsequent sections) for a comparison and step-by-step walkthrough of procedures for FDT, CDT, and EDT. One of the advantages we claim for FDT over CDT and EDT is that it doesn't require ad-hoc tailoring for different dilemmas (e.g., ad-hoc precommitment methods or ratification procedures, or modifications to the agent's prior).

The main thing that distinguishes FDT from CDT is how the true operator mentioned above functions. As far as I'm aware this is always inserted by hand. This is easy to for situations where entities make perfect simulations of one another, but there aren't even rough guidelines for what to do when the computations that are done cannot be delineated in such a clean manner. In addition, if this was a rich research field I would expect more "math that bites back", i.e., substantive results that reduce to clearly-defined mathematical problems whose result wasn't expected during the formalization.

This point about "load-bearing elements" is at its root an intuitive judgement that might be difficult for me to convey properly.

comment by Rob Bensinger (RobbBB) · 2017-11-13T18:26:20.648Z · LW(p) · GW(p)

The research directions that people at MIRI prioritize are already pretty heavily informed by work that was first developed or written up by people like Paul Christiano (at OpenAI), Stuart Armstrong (a MIRI research associate, primary affiliation FHI), Wei Dai, and others. MIRI researchers work on the things that look most promising to them, and those things get added to our agenda if they aren't already on the agenda.

Different researchers at MIRI have different ideas about what's most promising; e.g., the AAMLS agenda incorporated a lot of problems from Paul Christiano's agenda, and reflected the new intuitions and inside-view models brought to the table by the researchers who joined MIRI in early and mid-2015.

I'm guessing our primary disagreement is about how promising various object-level research directions at MIRI are. It might also be that you're thinking that there's less back-and-forth between MIRI and researchers at other institutions than actually occurs, or more viewpoint uniformity at MIRI than actually exists. Or you might be thinking that people at MIRI working on similar research problems together reflects top-down decisions by Eliezer, rather than reflecting (a) people with similar methodologies and intuitions wanting to work together, and (b) convergence happening faster between people who share the same physical space.

In this case, I think some of the relevant methodology/intuition differences are on questions like:

  • Are we currently confused on a fundamental level about what general-purpose good reasoning in physical environments is? Not just "how can we implement this in practice?", but "what (at a sufficient level of precision) are we even talking about?"
  • Can we become much less confused, and develop good models of how AGI systems decompose problems into subproblems, allocate cognitive resources to different subproblems, etc.?
  • Is it a top priority for developers to go into large safety-critical software projects like this with as few fundamental confusions about what they're doing as possible?

People who answer "yes" to those questions tend to cluster together and reach a lot of similar object-level conclusions, and people who answer "no" form other clusters. Resolving those very basic disagreements is therefore likely to be especially high-value.

I don't think Newcombe-like dilemmas are relevant for the reasoning of potentially dangerous AIs

The primary reason to try to get a better understanding of realistic counterfactual reasoning (e.g., what an agent's counterfactuals should look like in a decision problem) is that AGI is in large part about counterfactual reasoning. A generating methodology for a lot of MIRI researchers' work is that we want to ensure the developers of early AGI systems aren't "flying blind" with respect to how and why their systems work; we want developers to be able to anticipate the consequences of many design choices before they make them.

The idea isn't that AGI techniques will look like decision theory, any more than they'll look like probability theory. The idea is rather that it's essential to have a basic understanding of what decision-making and probabilistic reasoning are before you build a general-purpose probabilistic reasoner and decision-maker. Newcomb's problem is important in that context primarily because it's one of the biggest anomalies in our current understanding of counterfactual reasoning. Zeroing in on anomalies in established theories and paradigms, and tugging on loose threads until we get a sense of why our theories break down at this particular point, is a pretty standard and productive approach in the sciences.

All that said, Newcomblike scenarios are ubiquitous in real life, and would probably be much more so for AGI systems. I'll say more about this in a second comment.

comment by megasilverfist · 2017-11-10T08:52:37.562Z · LW(p) · GW(p)

A bit off the main topic, but what was the Ketosylent recipe?

Replies from: RobbBB, malo
comment by Malo (malo) · 2017-11-13T18:38:36.212Z · LW(p) · GW(p)

You might also want to check out Ketolent.

comment by avturchin · 2017-11-09T22:11:12.188Z · LW(p) · GW(p)

I think somewhere in the previous writing EY said that by printing money a central bank could increase prices, but it occured to me that it may be not true, because the prices will simultaneously fall in a harder currnecy, and thus hyperinflaion is deflation.

It reminds me the time when we in Russia had hyperinflation in 1990s, which also was deflation in more real assets. That is, Russian central bank printed so much money, that the prices constantly rose. However, the exchange rate to dollar grew even quicker and dollar prices constantly fell. Everybody started using dollar prices, written beneath ruble prices. The economy becomes a complete mess and collapsed in 1998 after failed attempt to stabilize ruble exchange rate using government short term debt.

The same could happen if too much dollars will be printed in the US - it could result in the deflation in gold and bitcoin prices.

Even now a lot of left economists, like Glaziev, ask Russian central bank to increase the money supply and to lend the money to money-starved factories. However, it is also well known that if money will be lended to the factories, they will be immideately converted in dollars, and this will crash russian ruble exchange rate. That is why left economists urges to ban dollar and other foreign currencies in Russia, and only after it to start to print money. However, such drastic measures will kill the economy as investment will stop.

Replies from: DonGeddis, gbear605
comment by DonGeddis · 2017-11-12T18:45:04.726Z · LW(p) · GW(p)

You mostly seem to be noticing that there is a difference between the nominal economy (the numbers on the prices), and the real economy (what resources you can buy with currency). That's certainly true - but actually beside the point. Because the point is that inflation (actually, NGDP) below trend, causes "business-cycle" recessions. The reason is that many contracts (wages, debts) are written in nominal terms, and so when the value of the Unit of Account (function of money) changes in an unexpected way, these fixed nominal contracts don't adjust quickly enough. The result is disequilibrium, rising unemployment and bankruptcies, etc. The fix is to keep nominal prices rising on trend.

Having exchange rates fall, or gold or asset prices rise, is an independent thing. It only matters if the currency begins to be abandoned (as you suggest with dollar prices in Russia), and especially if wage and debt contracts begin to be written in a different Unit of Account. It is stability in the value of the Unit of Account which affects macroeconomic stability.

Summary: "printing money could increase prices" is the whole point, and it doesn't matter if "prices fall in a harder currency" or there is "deflation in gold and bitcoin prices". As long as the local currency is the Unit of Account, then changes in the value of the local currency (aka local currency aggregate demand) are what matter.

Replies from: avturchin
comment by avturchin · 2017-11-15T20:10:49.970Z · LW(p) · GW(p)

As soon as agents start to realise that you started to print money, they begin to change their contracts into a harder currency. It has happened in Russia in 1990s, as all contracts were in dollars, because everybody was afraid that the governemnt will print more money. Government fought back, by banning use of foreign currency names in contrats. People created "artifical units", and everybody knows that any "artificial unit" in a contract = 1 USD. So one could increase price by printing money obly in small extent, as it undermines the believe of agents in your currency, and they will stop to use it.

comment by gbear605 · 2017-11-09T23:16:57.879Z · LW(p) · GW(p)

I believe that EY's main point with printing money is that central banks tend to not print enough money, even if some banks - such as Russia's in the 90's - print too much.

Just because you can do too much of something, it doesn't mean that you shouldn't do that thing more.

Replies from: avturchin, avturchin
comment by avturchin · 2017-11-10T10:20:56.351Z · LW(p) · GW(p)

Yes, it looks like Western central banks try a Goldilock approach - to print enough money to start inflation but not enough to cause catastrophic consequences. However, nobody knows when the consequences will be become catastrophic as it is non-lineary event, when people lose the faith in the money stability and start to change all of them into a harder currency.

Replies from: jsalvatier
comment by jsalvatier · 2017-11-10T17:52:42.389Z · LW(p) · GW(p)

That is not how it works.

Replies from: edward-jones, avturchin
comment by Edward Jones (edward-jones) · 2017-11-12T14:55:49.651Z · LW(p) · GW(p)

How does it work instead? I don't know much about this and your response doesn't seem to explain why you think that or what you think happens instead.

Replies from: avturchin
comment by avturchin · 2017-11-15T21:16:16.597Z · LW(p) · GW(p)

Basically, the idea is to print not too much and not too litle money, and pass between Scilla and Haribda of inflation and deflation. Also, the idea is to adjsut monetary policy of a central bank to act opposite to the business cycle, that is, to print more money during deflational recession and less during boom phase of economy. I have read about the idea in Dr.Roubini Econoblog during 2008 recession, and, in fact, it was presented as ideal policy, a dream, which ability to work is not known, but the dream is attractive to central banks.

If we look closer, we will find that we live in the period of unprecidented money printing, started after 2008. US government increased M1 (cash) money supply several times after 2008 - from around 1 trillion to 3.5 trillion in circulating bills. But it didn't produce "good" inflation, as these money pured into speculative assets. (This is where economists start to disagree, and my point become more speculative.)

It also good to note that while official inflation is very low, the price of a lot of important things increased several times, including gold, education, housing (for a longer period given previous bubble), stock prices, medicine, phones and even TV sets (but it is not presented in official infltion data because of "hedonistic indexing").

comment by avturchin · 2017-11-11T12:20:07.957Z · LW(p) · GW(p)

comment by avturchin · 2017-11-10T09:51:25.101Z · LW(p) · GW(p)

He wrote:

"Q. What if I say I'll print ten dollars and the market still thinks that's not enough?

A. Create even more money. Look, imagine creating a quadrillion dollars. Prices would go up then, right? I mean, a 12-year-old raised by goldbugs could understand that part… uh, it's possible you might need to add a 12-year-old raised by goldbugs to your advisory staff."

It look like he thinks that after printing a quadilion dollars, it will increase prices, but in fact only nominal prices will grow, but the prices in a harder currency will fail, and the market will be able to find the harder currency. For decades it was gold, but after 1970s central banks did exacltly what EY advises - they incresed fiat money supply, while playing against gold. There was elaborated scheme for lowering gold price by using gold lending and virtual gold, but it failed around 2008, and the price of gold jumped from 300 to 2000.

Replies from: jsalvatier
comment by jsalvatier · 2017-11-10T17:53:34.249Z · LW(p) · GW(p)

The point was to raise nominal prices in the first place

Replies from: avturchin
comment by avturchin · 2017-11-11T12:23:51.230Z · LW(p) · GW(p)

But raising nominal prices is economically useless if it is not the healthy inflation.

One could add 000 behind each prices number (thus 10 will be 10 000), something similar to denomination, but in the other diretion - thus nominal prices will grow 1000 times, but it will not affect economy.

Replies from: DonGeddis, Decius
comment by DonGeddis · 2017-11-12T18:51:24.795Z · LW(p) · GW(p)

"But raising nominal prices is economically useless" No, that's not true. Raising nominal prices helps with sticky wages & debts. Failing to raise nominal prices causes recession, unemployment, and bankruptcies.

"the healthy inflation" That phrase doesn't refer to anything. "Healthy" isn't a modifier that applies to "inflation". There is only one single thing: the change in the overall price level. There aren't "healthy" and "non-healthy" versions of that one thing.

"One could add 000 behind each prices number" No, you're likely thinking of a different hypothetical, something like an overnight currency devaluation. In those cases, wage and debt contracts are simultaneously converted from the "old peso" into the "new peso". That's a very, very different macroeconomic event. "Printing money", on the other hand, changes the prices of goods ... but sticky wage and debt contracts are unaffected (and thus devalued). It is exactly the fact that only some but not all prices are changed by central bank money printing, that causes "raising nominal prices" to have an effect on the real economy. (Exactly as you suggest, if all prices changed simultaneously, the real economy would be unaffected. It's important to understand that central bank money printing is very different.)

Replies from: avturchin
comment by avturchin · 2017-11-15T20:21:06.751Z · LW(p) · GW(p)

I mean by the "health level of inflation" the level of inflation which is benefitial to the economy without destoying belief in your currency, or creating an assets bubbles. As I explained in another comment below, printing money destoys contracts as people start to rewrite these contracts in a harder currency, as it happened in Russia during money printing experiments. The contracts were rewritten in dollars, exactly because russian central bank could not print dollars. As a result, the central bank lost the ability to affect inflation in dollars contracts. It had to pay a lot later to return the people beilef in russian ruble, by constntly manipulating currency rate.

comment by Decius · 2017-11-11T17:45:29.577Z · LW(p) · GW(p)

Nominal GDP also increases by 1000 times, and everyone's currency savings increases by 1k-fold, but the things which are explictly in nominal currency rather than in notes will keep the same number. The effect would be to destroy people who plan on using payments from debtors to cover future expenses, in the same way they would as if their debtors defaulted and paid only one part in a thousand of the debt, but without any default occuring.

Replies from: avturchin
comment by avturchin · 2017-11-15T20:22:17.181Z · LW(p) · GW(p)

And is it good?