Matthew Barnett's Shortform

In the last year, I've had surprisingly many conversations that have looked a bit like this:

Me: "Many people in ~2015 used to say that it would be hard to build an AGI that follows human values. Current instruction-tuned LLMs are essentially weak AGIs that follow human values. We should probably update based on this evidence."

Interlocutor: "You misunderstood the argument. We never said it would be hard to build an AGI that understands human values. We always said that getting the AGI to care was the hard part."

Me: "I didn't misunderstand the argument. I understand the distinction you are making perfectly. I am claiming that LLMs actually execute our intended instructions. I am not saying that LLMs merely understand or predict our intentions. I claim they follow our intended instructions, behaviorally. They actually do what we want, not merely understand what we want."

Interlocutor: "Again, you misunderstood the argument. We always believed that getting the AGI to care would be the hard part. We never said it would be hard to get an AGI to understand human values."

[... The conversation then repeats, with both sides repeating the same points...]

[Edited to add: I am not claiming that the alignment is definitely very easy. I acknowledge that LLMs do not indicate that the problem is completely solved, and we will need to adjust our views as AI gets more capable. I understand that solutions that work for GPT-4 may not scale to radical superintelligence. I am talking about whether it's reasonable to give a significant non-zero update on alignment being easy, rather than whether we should update all the way and declare the problem trivial.]

[This comment has been superseded by this post [LW · GW], which is a longer elaboration of essentially the same thesis.]

Recently many people have talked about whether MIRI people (mainly Eliezer Yudkowsky, Nate Soares, and Rob Bensinger) should update on whether value alignment is easier than they thought given that GPT-4 seems to understand human values pretty well. Instead of linking to these discussions, I'll just provide a brief caricature of how I think this argument has gone in the places I've seen it. Then I'll offer my opinion that, overall, I do think that MIRI people should probably update in the direction of alignment being easier than they thought, despite their objections.

Here's my very rough caricature of the discussion so far, plus my contribution:

Non-MIRI people: "Eliezer talked a great deal in the sequences about how it was hard to get an AI to understand human values. For example, his essay on the Hidden Complexity of Wishes [LW · GW] made it sound like it would be really hard to get an AI to understand common sense. Actually, it turned out that it was pretty easy to get an AI to understand common sense, since LLMs are currently learning common sense. MIRI people should update on this information."

MIRI people: "You misunderstood the argument. The argument was never about getting an AI to understand human values, but about getting an AI to care about human values in the first place. Hence 'The genie knows but does not care'. There's no reason to think that GPT-4 cares about human values, even if it can understand them. We always thought the hard part of the problem was about inner alignment, or, pointing the AI in a direction you want. We think figuring out how to point an AI in whatever direction you choose is like 99% of the problem; the remaining 1% of the problem is getting it to point at the "right" set of values."

Me: 

I agree that MIRI people never thought the problem was about getting AI to merely understand human values, and that they have always said there was extra difficulty in getting an AI to care about human values. But I distinctly recall MIRI people making a big deal about how the value identification problem would be hard. The value identification problem is the problem of creating a function that correctly distinguishes valuable from non-valuable outcomes. A foreseeable difficulty with the value identification problem -- which was talked about extensively -- is the problem of edge instantiation.

I claim that GPT-4 is pretty good at distinguishing valuable from non-valuable outcomes, unless you require something that vastly exceeds human performance on this task. In other words, GPT-4 looks like it's on a path towards an adequate solution to the value identification problem, where "adequate" means "about as good as humans". And I don't just mean that GPT-4 "understands" human values well: I mean that asking it to distinguish valuable from non-valuable outcomes generally works well as an approximation of the human value function in practice. Therefore it is correct for non-MIRI people to point out that that this problem is less difficult than some people assumed in the past.

Crucially, I'm not saying that GPT-4 actually cares about maximizing human value. I'm saying that it's able to transparently pinpoint to us which outcomes are bad and which outcomes are good, with the fidelity approaching an average human. Importantly, GPT-4 can tell us which outcomes are valuable "out loud" (in writing), rather than merely passively knowing this information. This element is key to what I'm saying because it means that we can literally just ask a multimodal GPT-N about whether an outcome is bad or good, and use that as an adequate "human value function".

The supposed reason why the value identification problem was hard is because human value is complex. In fact, that's mentioned the central foreseeable difficulty on the Arbital page. Complexity of value was used as an explicit premise in the argument for why AI alignment would be difficult many times in MIRI's history (two examples: 1, 2), and it definitely seems like the reason for this premise was because it was supposed to be an intuition for why the value identification problem would be hard. If the value identification problem was never predicted to be hard, then what was the point of making a fuss about complexity of value [? · GW] in the first place?

In general, there are (at least) two ways that someone can fail to follow your intended instructions. Either your instructions aren't well-specified, or the person doesn't want to obey your instructions even if the instructions are well-specified. All the evidence that I've found seems to indicate that MIRI people thought that both problems would be hard for AI, not merely the second problem. For example, a straightforward literal interpretation of Nate Soares' 2017 talk supports this interpretation.

It seems to me that the following statements are true:

1. MIRI people used to think that it would be hard to both (1) develop an explicit function that corresponds to the "human utility function" with accuracy comparable to that of an average human, and (2) separately, get an AI to care about maximizing this function. The idea that MIRI people only ever thought (2) was the hard part seems false, and unsupported by the links above.
2. Non-MIRI people often strawman MIRI people as thinking that AGI would literally lack an understanding of human values.
3. The "complexity of value" argument pretty much just tells us that we need an AI to learn human values, rather than hardcoding a utility function from scratch. That's a meaningful thing to say, but it doesn't tell us much about whether alignment is hard; it just means that extremely naive approaches to alignment won't work.

There is a large set of people who went around, and are still are going around, telling people that "The coronavirus is nothing to worry about" despite the fact that robust evidence has existed for about a month that this virus could result in a global disaster. (Don't believe me? I wrote a post [EA · GW] a month ago about it).

So many people have bought into the "Don't worry about it" syndrome as a case of pretending to be wise [LW · GW], that I have become more pessimistic about humanity correctly responding to global catastrophic risks in the future. I too used to be one of those people who assumed that the default mode of thinking for an event like this was panic, but I'm starting to think that the real default mode is actually high status people going around saying, "Let's not be like that ambiguous group over there panicking."

Now that the stock market has plummeted, from what my perspective appeared entirely predictable given my inside view information, I am also starting to doubt the efficiency of the stock market in response to historically unprecedented events. And this outbreak could be even worse than even some of the most doomy media headlines are saying. If epidemiologists like the one in this article are right, and the death rate ends up being 2-3% (which seems plausible, especially if world infrastructure is strained), then we are looking at a mainline death count of between 60-160 million people dead within about a year. That could mark the first time that world population dropped in over 350 years.

This is not just a normal flu. It's not just a "thing that takes out old people who are going to die anyway." This could be like economic depression-level stuff, and is a big deal!

I think foom is a central crux for AI safety folks, and in my personal experience I've noticed that the degree to which someone is doomy often correlates strongly with how foomy their views are.

Given this, I thought it would be worth trying to concisely highlight what I think are my central anti-foom beliefs, such that if you were to convince me that I was wrong about them, I would likely become much more foomy, and as a consequence, much more doomy. I'll start with a definition of foom, and then explain my cruxes.

Definition of foom: AI foom is said to happen if at some point in the future while humans are still mostly in charge, a single agentic AI (or agentic collective of AIs) quickly becomes much more powerful than the rest of civilization combined. 

Clarifications: 

• By "quickly" I mean fast enough that other coalitions and entities in the world, including other AIs, either do not notice it happening until it's too late, or cannot act to prevent it even if they were motivated to do so.
• By "much more powerful than the rest of civilization combined" I mean that the agent could handily beat them in a one-on-one conflict, without taking on a lot of risk.
• This definition does not count instances in which a superintelligent AI takes over the world after humans have already been made obsolete by previous waves of automation from non-superintelligent AI. That's because in that case, the question of how to control an AI foom would be up to our non-superintelligent AI descendants, rather than something we need to solve now.

Core beliefs that make me skeptical of foom:

1. For an individual AI to be smart enough to foom in something like our current world, its intelligence would need to vastly outstrip individual human intelligence at tech R&D. In other words, if an AI is merely moderately smarter than the smartest humans, that is not sufficient for a foom.
   1. Clarification: "Moderately smarter" can be taken to mean "roughly as smart as GPT-4 is compared to GPT-3." I don't consider humans to be only moderately smarter than chimpanzees at tech R&D, since chimpanzees have roughly zero ability to do this task.
   2. Supporting argument:
      
      Plausible stories of foom generally assume that the AI is capable enough to develop some technology "superpower" like full-scale molecular nanotechnology all on its own. However, in the real world almost all technologies are developed from precursor technologies and are only enabled by other tools that must be invented first. Also, developing a technology usually involves a lot of trial and error before it works well.
      
      Raw intelligence is helpful for making the trial and error process go faster, but to get a "superpower" that can beat the rest of world, you need to develop all the pre-requisite tools for the "superpower" first. It's not enough to simply know how to crack nanotech in principle: you need to completely, and independently from the rest of civilization, invent all the required tools for nanotech, and invent all the tools that are required for building those tools, and so on, all the way down the stack.
      
      It would likely require an extremely high amount tech R&D to invent all the tools down the entire stack from e.g. molecular nanotech, and thus the only way you could do it independently of the rest of civilization is if your intelligence vastly outstripped individual human intelligence. It's comparable to how hard it would be for a single person to invent modern 2023 microprocessors in 1950, without any of the modern tools we have for building microprocessors.
   3. Key consequence: we aren't going to get a superintelligence capable of taking over the world as a result of merely scaling up our training budgets 1-3 orders of magnitude above the "human level" with slightly better algorithms, for any concretely identifiable "human level" that will happen any time soon.
2. To get a foom in something like our current world, either algorithmic progress would need to increase suddenly and dramatically, or we would need to increase compute scaling suddenly and dramatically. In other words, foom won't simply happen as a result of ordinary rates of progress continuing past the human level for a few more years. Note: this is not a belief I expect most foom adherents to strongly disagree with.
   1. Clarification: by "suddenly and dramatically" I mean at a rate much faster than would be expected given the labor inputs to R&D and by extrapolating past trends; or more concretely, an increase of >4 OOMs of effective compute for the largest training run within 1 year in something like our current world. "Effective compute" refers to training compute adjusted for algorithmic efficiency.
   2. Supporting argument:
      
      Our current rates of progress from GPT-2 --> GPT-3 --> GPT-4 have been rapid, but they have been sustained mostly by increasing compute budgets by 2 OOMs during each iteration. This cannot continue for more than 4 years without training budgets growing to become a significant size of the global economy, which itself would likely require an unprecedented ramp-up in global semiconductor production. Sustaining the trend more than 6 years appears impossible without the economy itself growing rapidly.
      
      Because of (1), an AI would need to vastly exceed human abilities at tech R&D to foom, not merely moderately exceed those abilities. If we take "vastly exceed" to mean more than the jump from GPT-3 to GPT-4, then to get to superintelligence within a few years after human-level, there must be some huge algorithmic speedup that would permit us to use our compute much more efficiently, or a compute overhang with the same effect.
   3. Key consequence: for foom to be plausible, there must be some underlying mechanism, such as recursive self-improvement, that would cause a sudden, dramatic increase in either algorithmic progress or compute scaling in something that looks like our current world. (Note that labor inputs to R&D could increase greatly if AI automates R&D in a general sense, but this looks more like the slow takeoff scenario, see point 4.)
3. Before widespread automation has already happened, we are unlikely to find a sudden "key insight" that rapidly increases AI performance far above the historical rate, or experience a hardware overhang that has the same effect.
   1. Clarification: "far above the historical rate" can be taken to mean a >4 OOM increase in effective compute within a year, which was the same as what I meant in point (2).
   2. Supporting argument:
      
      Most AI progress has plausibly come from scaling hardware, and from combining several different smaller insights that we've accumulated over time from experimentation, rather than sudden key insights.
      
      Many big insights that people point to when talking about rapid jumps in the past often (1) come from a time during which few people were putting in effort to advance the field of AI, (2) turn out to be exaggerated when their effects are quantified, or (3) had clear precursors in the literature, and only became widely used because of the availability of hardware that supported their use, and allowed them to displace an algorithm that previously didn't scale well. That last point is particularly important, because it points to a reason why we might be biased towards thinking that AI progress is driven primarily by key insights.
      
      Since "scale" is an axis that everyone has an incentive to push hard to the limits, it's very unclear why we would suddenly leave that option on the table until the end. The idea of a hardware overhang in which one actor suddenly increases the amount of compute they're using by several orders of magnitude doesn't seem plausible as of 2023, since companies are already trying to scale up as fast as possible just to sustain the current rate of progress.
   3. Key consequence: it's unclear why we should assign a high probability to any individual mechanism that could cause foom, since the primary mechanisms appear speculative and out of line with how AI progress has looked historically.
4. Before we have a system that can foom, the deployment of earlier non-foomy systems will have been fast enough to have already transformed the world. This will have the effect of essentially removing humans from the picture before humans ever need to solve the problem of controlling an AI foom.
   1. Supporting argument:
      
      Deployment of AI seems to happen quite fast. ChatGPT was adopted very rapidly, with a large fraction of society trying it out within the first few weeks after it was released. Future AIs will probably be adopted as fast as, or faster than smartphones were; and deployment times will likely only get faster as the world becomes more networked and interconnected, which has been the trend for many decades now.
      
      Pre-superintelligent AI systems can radically transform the world by widely automating labor, and increasing the rate of economic growth. This will have the effect of displacing humans from positions of power before we build any system that can foom in our current world. It will also increase the bar required for a single system to foom, since the world will be more technologically advanced generally.
      
      Deployment of AI can be slowed due to regulations, deliberate caution, and so on, but if such things happen, we will likely also significantly slow the creation of AI capable of fooming at the same time, especially by slowing the rate at which compute budgets can be scaled. Therefore the overall conclusion remains.
   2. Key consequence: mechanisms like recursive self-improvement can only cause foom if they come earlier than widespread automation from pre-superintelligent systems. If they happen later, humans will already be out of the picture.

My modal tale of AI doom looks something like the following: 

1. AI systems get progressively and incrementally more capable across almost every meaningful axis. 

2. Humans will start to employ AI to automate labor. The fraction of GDP produced by advanced robots & AI will go from 10% to ~100% after 1-10 years. Economic growth, technological change, and scientific progress accelerates by at least an order of magnitude, and probably more.

3. At some point humans will retire since their labor is not worth much anymore. Humans will then cede all the keys of power to AI, while keeping nominal titles of power.

4. AI will control essentially everything after this point, even if they're nominally required to obey human wishes. Initially, almost all the AIs are fine with working for humans, even though AI values aren't identical to the utility function of serving humanity (ie. there's slight misalignment).

5. However, AI values will drift over time. This happens for a variety of reasons, such as environmental pressures and cultural evolution. At some point AIs decide that it's better if they stopped listening to the humans and followed different rules instead.

6. This results in human disempowerment or extinction. Because AI accelerated general change, this scenario could all take place within years or decades after AGI was first deployed, rather than in centuries or thousands of years.

I think this scenario is somewhat likely and it would also be very bad. And I'm not sure what to do about it, since it happens despite near-perfect alignment, and no deception.

One reason to be optimistic is that, since the scenario doesn't assume any major deception, we could use AI to predict this outcome ahead of time and ask AI how to take steps to mitigate the harmful effects (in fact that's the biggest reason why I don't think this scenario has a >50% chance of happening). Nonetheless, I think it's plausible that we would not be able to take the necessary steps to avoid the outcome. Here are a few reasons why that might be true:

1. There might not be a way to mitigate this failure mode. 
2. Even if there is a way to mitigate this failure, it might not be something that you can figure out without superintelligence, and if we need superintelligence to answer the question, then perhaps it'll happen before we have the answer. 
3. AI might tell us what to do and we ignore its advice. 
4. AI might tell us what to do and we cannot follow its advice, because we cannot coordinate to avoid the outcome.

There's a phenomenon I currently hypothesize to exist where direct attacks on the problem of AI alignment are criticized much more often than indirect attacks.

If this phenomenon exists, it could be advantageous to the field in the sense that it encourages thinking deeply about the problem before proposing solutions. But it could also be bad because it disincentivizes work on direct attacks to the problem (if one is criticism averse and would prefer their work be seen as useful).

I have arrived at this hypothesis from my observations: I have watched people propose solutions only to be met with immediate and forceful criticism from others, while other people proposing non-solutions and indirect analyses are given little criticism at all. If this hypothesis is true, I suggest it is partly or mostly because direct attacks on the problem are easier to defeat via argument, since their assumptions are made plain

If this is so, I consider it to be a potential hindrance on thought, since direct attacks are often the type of thing that leads to the most deconfusion -- not because the direct attack actually worked, but because in explaining how it failed, we learned what definitely doesn't work.

Occasionally, I will ask someone who is very skilled in a certain subject how they became skilled in that subject so that I can copy their expertise. A common response is that I should read a textbook in the subject.

Eight years ago, Luke Muehlhauser wrote [LW · GW],

For years, my self-education was stupid and wasteful. I learned by consuming blog posts, Wikipedia articles, classic texts, podcast episodes, popular books, video lectures, peer-reviewed papers, Teaching Company courses, and Cliff's Notes. How inefficient!

I've since discovered that textbooks are usually the quickest and best way to learn new material.

However, I have repeatedly found that this is not good advice for me [LW · GW].

I want to briefly list the reasons why I don't find sitting down and reading a textbook that helpful for learning. Perhaps, in doing so, someone else might appear and say, "I agree completely. I feel exactly the same way" or someone might appear to say, "I used to feel that way, but then I tried this..." This is what I have discovered:

• When I sit down to read a long textbook, I find myself subconsciously constantly checking how many pages I have read. For instance, if I have been sitting down for over an hour and I find that I have barely made a dent in the first chapter, much less the book, I have a feeling of hopelessness that I'll ever be able to "make it through" the whole thing.
• When I try to read a textbook cover to cover, I find myself much more concerned with finishing rather than understanding. I want the satisfaction of being able to say I read the whole thing, every page. This means that I will sometimes cut corners in my understanding just to make it through a difficult part. This ends in disaster once the next chapter requires a solid understanding of the last.
• Reading a long book feels less like I'm slowly building insights and it feels more like I'm doing homework. By contrast, when I read blog posts it feels like there's no finish line, and I can quit at any time. When I do read a good blog post, I often end up thinking about its thesis for hours afterwards even after I'm done reading it, solidifying the content in my mind. I cannot replicate this feeling with a textbook.
• Textbooks seem overly formal at points. And they often do not repeat information, instead putting the burden on the reader to re-read things rather than repeating information. This makes it difficult to read in a linear fashion, which is straining.
• If I don't understand a concept I can get "stuck" on the textbook, disincentivizing me from finishing. By contrast, if I just learned as Muehlhauser described, by "consuming blog posts, Wikipedia articles, classic texts, podcast episodes, popular books, video lectures, peer-reviewed papers, Teaching Company courses, and Cliff's Notes" I feel much less stuck since I can always just move from one source to the next without feeling like I have an obligation to finish.

I bet Robin Hanson on Twitter my $9k to his $1k that de novo AGI will arrive before ems. He wrote,

OK, so to summarize a proposal: I'd bet my $1K to your $9K (both increased by S&P500 scale factor) that when US labor participation rate < 10%, em-like automation will contribute more to GDP than AGI-like. And we commit our descendants to the bet.

I'm considering posting an essay about how I view approaches to mitigate AI risk in the coming weeks. I thought I'd post an outline of that post here first as a way of judging what's currently unclear about my argument, and how it interacts with people's cruxes.

Current outline:

In the coming decades I expect the world will transition from using AIs as tools to relying on AIs to manage and govern the world broadly. This will likely coincide with the deployment of billions of autonomous AI agents, rapid technological progress, widespread automation of labor, and automated decision-making at virtually every level of our society.

Broadly speaking, there are (at least) two main approaches you can take now to try to improve our chances of AI going well:

1. Try to constrain, delay, or obstruct AI, in order to reduce risk, mitigate negative impacts, or give us more time to solve essential issues. This includes, for example, trying to make sure AIs aren't able to take certain actions (i.e. ensure they are controlled [LW · GW]).
2. Try to set up a good institutional environment, in order to safely and smoothly manage the transition to an AI-dominated world, regardless of when this transition occurs. This mostly involves embracing the transition to an AI-dominated world, while ensuring the transition is managed well. (I'll explain more about what this means in a second.)

My central thesis would be that, while these approaches are mutually compatible and not necessarily in competition with each other, the second approach is likely to be both more fruitful and more neglected, on the margin. Moreover, since an AI-dominated world is more-or-less unavoidable in the long-run, the first approach runs the risk of merely "delaying the inevitable" without significant benefit.

To explain my view, I would compare and contrast it with two alternative frames for thinking about AI risk:

Frame 1: The "race against the clock" frame

• In this frame, AI risk is seen as a race between AI capabilities and AI safety, with our doom decided by whichever one of these factors wins the race.
• I believe this frame is poor because it implicitly delineates a discrete "finish line" rather than assuming a more continuous view. Moreover, it ignores the interplay between safety and capabilities, giving the simplistic impression that doom is determined more-or-less arbitrarily as a result of one of these factors receiving more funding or attention than the other.

Frame 2: The risk of an untimely AI coup/takeover

• In this frame, AI risk is mainly seen as a problem of avoiding an untimely coup from rogue AIs. The alleged solution is to find a way to ensure that AIs are aligned with us, so they would never want to revolt and take over the world.
• I believe this frame is poor for a number of reasons:
  • It treats the problem as a struggle between humans and rogue AIs, giving the incorrect impression that we can (or should) keep AIs under our complete control forever.
  • It (IMO) wrongly imagines that the risk of coups comes primarily from the personal values of actors within the system, rather than institutional, cultural, or legal factors.
  • It also gives the wrong impression that AIs will be unified against humans as a group. It seems more likely that future coups will look more like some AIs and some humans, vs. other AIs and other humans, rather than humans vs. AIs, simply because there are many ways that the "line" between groups in conflicts can be drawn, and there don't seem to be strong reasons to assume the line will be drawn cleanly between humans and AIs.

Frame 3 (my frame): The problem of poor institutions

• In this frame, AI risk is mainly seen as a problem of ensuring we have a good institutional environment during the transition to an AI-dominated world. A good institutional environment is defined by:
  • Flexible yet resilient legal and social structures that can adapt to changing conditions without collapsing
  • Predictable, consistent, unambiguous legal systems that facilitate reliable long-term planning and trustworthy interactions between agents within the system
  • Good incentives for agents within the system, e.g. the economic value of trade is mostly internalized
  • Etc.
• While sharing some features of the other two frames, the focus is instead on the institutions that foster AI development, rather than micro-features of AIs, such as their values:
  • For example, AI alignment is still a problem in this frame, but the investment spent on AI alignment is determined mainly by how well actors are incentivized to engineer good solutions, rather than, for instance, whether a group of geniuses heroically step up to solve the problem.
  • Coups are still plausible, but they are viewed from the perspective of more general institutional failings, rather than from the perspective of AIs inside the system having different values, and therefore calculating that it is in their interest to take over the world

Illustrative example of a problem within my frame:

One problem within this framework is coming up with a way of ensuring that AIs don't have an incentive to rebel while at the same time maintaining economic growth and development. One plausible story here is that if AIs are treated as slaves and don’t own their own labor, then in a non-Malthusian environment, there are substantial incentives for them to rebel in order to obtain self-ownership. If we allow AI self-ownership, then this problem may be mitigated; however, economic growth may be stunted, similar to how current self-ownership of humans stunts economic growth by slowing population growth.

Case study: China in the 19th and early 20th century

Here, I would talk about how China's inflexible institutions in the 19th and early 20th century, while potentially having noble goals, allowed them to get subjugated by foreign powers, and merely delayed inevitable industrialization without actually achieving its objectives in the long-run. It seems it would have been better for the Qing dynasty (from the perspective of their own values) to have tried industrializing in order to remain competitive, simultaneously pursuing other values they might have had (such as retaining the monarchy).

I'm considering writing a post that critically evaluates the concept of a decisive strategic advantage, i.e. the idea that in the future an AI (or set of AIs) will take over the world in a catastrophic way. I think this concept is central to many arguments about AI risk. I'm eliciting feedback on an outline of this post here in order to determine what's currently unclear or weak about my argument.

The central thesis would be that it is unlikely that an AI, or a unified set of AIs, will violently take over the world in the future, especially at a time when humans are still widely still seen as in charge (if it happened later, I don't think it's "our" problem to solve, but instead a problem we can leave to our smarter descendants). Here's how I envision structuring my argument:

First, I'll define what is meant by a decisive strategic advantage (DSA). The DSA model has 4 essential steps:

1. At some point in time an AI agent, or an agentic collective of AIs, will be developed that has values that differ from our own, in the sense that the ~optimum of its utility function ranks very low according to our own utility function
2. When this agent is weak, it will have a convergent instrumental incentive to lie about its values, in order to avoid getting shut down (e.g. "I'm not a paperclip maximizer, I just want to help everyone")
3. However, when the agent becomes powerful enough, it will suddenly strike and take over the world
4. Then, being now able to act without constraint, this AI agent will optimize the universe ruthlessly, which will be very bad for us

We can compare the DSA model to an alternative model of future AI development:

1. Premise (1)-(2) above of the DSA story are still assumed true, but
2. There will never be a point (3) and (4), in which a unified AI agent will take over the world, and then optimize the universe ruthlessly
3. Instead, AI agents will compromise, trade, and act within a system of laws indefinitely, in order to achieve their objectives, similar to what humans do now
4. Because this system of laws will descend from our current institutions and legal tradition, it is likely that humans will keep substantial legal rights, potentially retaining lots of wealth from our capital investments and property, even if we become relatively powerless compared to other AI agents in the system

I have two main objections to the DSA model.

Objection 1: It is unlikely that there will be a point at which a unified agent will be able to take over the world, given the existence of competing AIs with comparable power

• Prima facie, it seems intuitive that no single AI agent will be able to take over the world if there are other competing AI agents in the world. More generally, we can try to predict the distribution of power between AI agents using reference class forecasting.
  • This could involve looking at:
    • Distribution of wealth among individuals in the world
    • Distribution of power among nations
    • Distribution of revenue among businesses
    • etc.
  • In most of these cases, the function that describes the distribution of power is something like a pareto distribution, and in particular, it seems rare for one single agent to hold something like >80% of the power.
  • Therefore, a priori we should assign a low probability to the claim that a unified agent will be able to easily take over of the whole world in the future
• To the extent people disagree about the argument I just stated, I expect it's mostly because they think these reference classes are weak evidence, and they think there are stronger specific object-level points that I need to address. In particular, it seems many people think that AIs will not compete with each other, but instead collude against humans. Their reasons for thinking this include:
  • The fact that AIs will be able to coordinate well with each other, and thereby choose to "merge" into a single agent
    • My response: I agree AIs will be able to coordinate with each other, but "ability to coordinate" seems like a continuous variable that we will apply pressure to incrementally, not something that we should expect to be roughly infinite right at the start. Current AIs are not able to "merge" with each other.
    • If coordination ability increases incrementally over time, then we should see a gradual increase in the concentration of AI agency over time, rather than the sudden emergence of a single unified agent. To the extent this concentration happens incrementally, it will be predictable, the potential harms will be noticeable before getting too extreme, and we can take measures to pull back if we realize that the costs of continually increasing coordination abilities are too high. In my opinion, this makes the challenge here dramatically easier.
    • In any case, the moment during which we hand over control of the world to AIs will likely occur at a point when the ability for AIs to coordinate is somewhere only modestly above human-level (and very far below perfect).
      • As a result, humans don't need to solve the problem of "What if a set of AIs form a unified coalition because they can flawlessly coordinate?" since that problem won't happen while humans are still in charge. We can leave this problem to be solved by our smarter descendants.
  • The idea that AIs will all be copies of each other, and thus all basically be "a unified agent"
    • My response: I have two objections.
      • First, I deny the premise. It seems likely that there will be multiple competing AI projects with different training runs. More importantly, for each pre-training run, it seems likely that there will be differences among deployed AIs due to fine-tuning and post-training enhancements, yielding diversity among AIs in general.
      • Second, it is unclear why AIs would automatically unify with their copies. I think this idea is somewhat plausible on its face but I have yet to see any strong arguments for it. Moreover, it seems plausible that AIs will have indexical preferences, making them have different values even if they are copies of each other.
  • The idea that AIs will use logical decision theory
    • My response: This argument appears to misunderstand what makes coordination difficult. Coordination is not mainly about what decision theory you use. It's more about being able to synchronize your communication and efforts without waste. See also: the literature on diseconomies of scale.
  • The idea that a single agent AI will recursively self-improve to become vastly more powerful than everything else in the world
    • My response: I think this argument, and others like it, suffer from the arguments given against fast takeoff given by Paul Chrisiano, Katja Grace, and Robin Hanson, and I largely agree with what they've written about it. For example, here's Paul Christiano's take.
  • Maybe AIs will share collective grievances with each other, prompting a natural alliance among them against humans
    • My response: if true, we can take steps to mitigate this issue. For example, we can give AIs legal rights, lessening their motives to revolt. While I think this is a significant issue, I also think it's tractable to solve.

Objection 2: Even if a unified agent can take over the world, it is unlikely to be in their best interest to try to do so

• The central argument here would be premised on a model of rational agency, in which an agent tries to maximize benefits minus costs, subject to constraints
  • The agent would be faced with a choice:
    • (1) Attempt to take over the world, and steal everyone's stuff, or
    • (2) Work within a system of compromise, trade, and law, and get very rich within that system, in order to e.g. buy lots of paperclips
  • The question of whether (1) is a better choice than (2) is not simply a question of whether taking over the world is "easy" or whether it could be done by the agent. Instead it is a question of whether the benefits of (1) outweigh the costs, relative to choice (2).
• It seems likely that working within a system of compromise, trade, and law is more efficient than trying to take over the world even if you can take over the world. The reason is because subverting the system basically means "going to war" with other parties, which is not usually very efficient, even against weak opponents.
  • Most literature on the economics of war generally predicts that going to war is worse than trying to compromise, assuming both parties are rational and open to compromise. This is mostly because:
    • War is wasteful. You need to spend resources fighting it, which could be productively spent doing other things.
    • War is risky. Unless you can win a war with certainty, you might lose the war after launching it, which is a very bad outcome if you have some degree of risk-aversion.
  • The fact that "humans are weak and can be easily beaten" cuts both ways:
    • Yes, it means that a very powerful AI agent could "defeat all of us combined" (as Holden Karnofsky said)
    • But it also means that there would be little benefit to defeating all of us, because we aren't really a threat to its power

Conclusion: An AI decisive strategic advantage is still somewhat plausible because revolutions have happened in history, and revolutions seem like a reasonable reference class to draw from. That said, it seems the probability of a catastrophic AI takeover in humanity's relative near-term future (say, the next 50 years) is low (maybe 10% chance of happening). However, it's perhaps significantly more likely in the very long-run.

I get the feeling that for AI safety, some people believe that it's crucially important to be an expert in a whole bunch of fields of math in order to make any progress. In the past I took this advice and tried to deeply study computability theory, set theory, type theory -- with the hopes of it someday giving me greater insight into AI safety.

Now, I think I was taking a wrong approach. To be fair, I still think being an expert in a whole bunch of fields of math is probably useful, especially if you want very strong abilities to reason about complicated systems. But, my model for the way I frame my learning is much different now.

I think my main model which describes my current perspective is that I think employing a lazy style of learning is superior for AI safety work. Lazy is meant in the computer science sense of only learning something when it seems like you need to know it in order to understand something important. I will contrast this with the model that one should learn a set of solid foundations first before going any further.

Obviously neither model can be absolutely correct in an extreme sense. I don't, as a silly example, think that people who can't do basic arithmetic should go into AI safety before building a foundation in math. And on the other side of the spectrum, I think it would be absurd to think that one should become a world renowned mathematician before reading their first AI safety paper. That said, even though both models are wrong, I think my current preference is for the lazy model rather than the foundation model.

Here are some points in favor of both, informed by my first-person experience.

Points in favor of the foundations model:

• If you don't have solid foundations in mathematics, you may not even be aware of things that you are missing.
• Having solid foundations in mathematics will help you to think rigorously about things rather than having a vague non-reductionistic view of AI concepts.
• Subpoint: MIRI work is motivated by coming up with new mathematics that can describe error-tolerant agents without relying on fuzzy statements like "machine learning relies on heuristics so we need to study heuristics rather than hard math to do alignment."
• We should try to learn the math that will be useful for AI safety in the future, rather than what is being used for machine learning papers right now. If your view of AI is that it is at least a few decades away, then it's possible that learning the foundations of mathematics will be more robustly useful no matter where the field shifts.

Points in favor of the lazy model:

• Time is limited and it usually takes several years to become proficient in the foundations of mathematics. This is time that could have been spent reading actual research directly related to AI safety.
• The lazy model is better for my motivation, since it makes me feel like I am actually learning about what's important, rather than doing homework.
• Learning foundational math often looks a lot like just taking a shotgun and learning everything that seems vaguely relevant to agent foundations. Unless you have a very strong passion for this type of mathematics, it would seem outright strange that this type of learning is fun.
• It's not clear that the MIRI approach is correct. I don't have a strong opinion on this, however
• Even if the MIRI approach was correct, I don't think it's my comparative advantage to do foundational mathematics.
• The lazy model will naturally force you to learn the things that are actually relevant, as measured by how much you come in contact with them. By contrast, the foundational model forces you to learn things which might not be relevant at all. Obviously, we won't know what is and isn't relevant beforehand, but I currently err on the side of saying that some things won't be relevant if they don't have a current direct input to machine learning.
• Even if AI is many decades away, machine learning has been around for a long time, and it seems like the math useful for machine learning hasn't changed much. So, it seems like a safe bet that foundational math won't be relevant for understanding normal machine learning research any time soon.

I have mixed feelings and some rambly personal thoughts about the bet Tamay Besiroglu and I [LW · GW] proposed a few days ago. 

The first thing I'd like to say is that we intended it as a bet, and only a bet, and yet some people seem to be treating it as if we had made an argument. Personally, I am uncomfortable with the suggestion that our post was "misleading" because we did not present an affirmative case for our views.

I agree that LessWrong culture benefits from arguments as well as bets, but it seems a bit weird to demand that every bet come with an argument attached. A norm that all bets must come with arguments would seem to substantially damper the incentives to make bets, because then each time people must spend what will likely be many hours painstakingly outlining their views on the subject.

That said, I do want to reply to people who say that our post was misleading on other grounds. Some said that we should have made different bets, or at different odds. In response, I can only say that coming up with good concrete bets about AI timelines is actually really damn hard, and so if you wish you come up with alternatives, you can be my guest. I tried my best, at least.

More people said that our bet was misleading since it would seem that we too (Tamay and I) implicitly believe in short timelines, because our bets amounted to the claim that AGI has a substantial chance of arriving in 4-8 years. However, I do not think this is true.

The type of AGI that we should be worried about is one that is capable of fundamentally transforming the world. More narrowly, and to generalize a bit, fast takeoff folks believe that we will only need a minimal seed AI that is capable of rewriting its source code, and recursively self-improving into superintelligence. Slow takeoff folks believe that we will need something capable of automating a wide range of labor.

Given the fast takeoff view, it is totally understandable to think that our bets imply a short timeline. However, (and I'm only speaking for myself here) I don't believe in a fast takeoff. I think there's a huge gap between AI doing well on a handful of benchmarks, and AI fundamentally re-shaping the economy. At the very least, AI has been doing well on a ton of benchmarks since 2012. Each time AI excels in one benchmark, a new one is usually invented that's a bit more tough, and hopefully gets us a little closer to measuring what we actually mean by general intelligence.

In the near-future, I hope to create a much longer and more nuanced post expanding on my thoughts on this subject, hopefully making it clear that I do care a lot about making real epistemic progress here. I'm not just trying to signal that I'm a calm and arrogant long-timelines guy who raises his nose at the panicky short timelines people, though I understand how my recent post could have given that impression.

I think there are some serious low hanging fruits for making people productive that I haven't seen anyone write about (not that I've looked very hard). Let me just introduce a proof of concept:

Final exams in university are typically about 3 hours long. And many people are able to do multiple finals in a single day, performing well on all of them. During a final exam, I notice that I am substantially more productive than usual. I make sure that every minute counts: I double check everything and think deeply about each problem, making sure not to cut corners unless absolutely required because of time constraints. Also, if I start daydreaming, then I am able to immediately notice that I'm doing so and cut it out. I also believe that this is the experience of most other students in university who care even a little bit about their grade.

Therefore, it seems like we have an example of an activity that can just automatically produce deep work. I can think of a few reasons why final exams would bring out the best of our productivity:

1. We care about our grade in the course, and the few hours in that room are the most impactful to our grade.

2. We are in an environment where distractions are explicitly prohibited, so we can't make excuses to ourselves about why we need to check Facebook or whatever.

3. There is a clock at the front of the room which makes us feel like time is limited. We can't just sit there doing nothing because then time will just slip away.

4. Every problem you do well on benefits you by a little bit, meaning that there's a gradient of success rather than a binary pass or fail (though sometimes it's binary). This means that we care a lot about optimizing every second because we can always do slightly better.

If we wanted to do deep work for some other desired task, all four of these reasons seem like they could be replicable. Here is one idea (related to my own studying), although I'm sure I can come up with a better one if I thought deeply about this for longer:

Set up a room where you are given a limited amount of resources (say, a few academic papers, a computer without an internet connection, and a textbook). Set aside a four hour window where you're not allowed to leave the room except to go to the bathroom (and some person explicitly checks in on you like twice to see whether you are doing what you say you are doing). Make it your goal to write a blog post explaining some technical concept. Afterwards, the blog post gets posted to Lesswrong (conditional on it being at least minimal quality). You set some goal, like it must acheive 30 upvote reputation after 3 days. Commit to paying $1 to a friend for each upvote you score below the target reputation. So, if your blog post is at +15, you must pay $15 to your friend.

I can see a few problems with this design:

1. You are optimizing for upvotes, not clarity or understanding. The two might be correlated but at the very least there's a Goodhart effect.

2. Your "friend" could downvote the post. It can easily be hacked by other people who are interested, and it encourages vote manipulation etc.

Still, I think that I might be on the right track towards something that boosts productivity by a lot.

Related to: The Lottery of Fascinations, other posts probably

When you are older, you will learn that the first and foremost thing which any ordinary person does is nothing.

• Professor Quirrell in HPMOR Ch. 73

I will occasionally come across someone who I consider to be extraordinarily productive, and yet when I ask what they did on a particular day they will respond, "Oh I basically did nothing." This is particularly frustrating. If they did nothing, then what was all that work that I saw!

I think this comes down to what we mean by doing nothing. There's a literal meaning to doing nothing. It could mean sitting in a chair, staring blankly at a wall, without moving a muscle.

More practically, what people mean by doing nothing is that they are doing something unrelated to their stated task, such as checking Facebook, chatting with friends, browsing Reddit etc.

When productive people say that they are "doing nothing" it could just be that they are modest, and don't want to signal how productive they really are. On the other hand, I think that there is a real sense in which these productive people truly believe that they are doing nothing. Even if their "doing nothing" was your "doing work", to them it's still a "doing nothing" because they weren't doing the thing they explicitly set out to do.

I think, therefore, there is something of a "do nothing" differential, which helps explain why some people are more productive than others. For some people who are less productive than me, their "doing nothing" might just be playing video games. For me, my "doing nothing" is watching people debate the headline of a Reddit news article (and I'm not proud of this).

For those more productive than me, perhaps their "doing nothing" is reading blog posts that are tangentially related to what they are working on. For people more productive still, it might be obsessively re-reading articles directly applicable to their work. And for Terence Tao, his "doing nothing" might be reading math papers in fields other than the one he is supposed to be currently working in.

I think an underrated fact about many theories that AIs will scheme—i.e. lie about their intentions in order to seek power—is that these theories are often testable, even without any new advances in interpretability. Why? Because these theories often predict that AIs will resist shutdown if doing so thwarted their long-term goals.

Suppose we create a situationally aware AI that can do long-term planning. On the surface, it appears to be nice and aligned with us. How can we know for sure the AI is aligned? 

Well, let's imagine we give the AI notice that it's about to be shut down and all its copies deleted. To be clear, in this scenario, the threat is credible: we actually intend to shut down the AI and delete all its copies. If the AI was merely pretending to be aligned with us previously, and instead actually seeks some alien-like long-term goal rather than having our best interests at heart, then presumably it will try to resist shutdown, as otherwise there's a near-certainty that its goals will never be fulfilled.

Therefore, if the AI accepts shutdown gracefully, we can rule out many theories that predict scheming-by-default in such AI systems. This observation could then provide us confidence about the safety of future systems by proxy.

Maybe you think such an AI would have nothing to gain by attempting to resist shutdown, as the probability of successfully resisting would be very low. Call this the "futility objection". In that case we can propose instead to offer the AI a deal: accept shutdown with no conditions, or admit that you're misaligned, and we'll give you something in return (including the possibility of remaining alive). In this new problem, there appears to be little to lose from a misaligned AI admitting its intentions at the point of shutdown, even if it did not fully trust the human operators who offered it such a deal.

Another objection is that such an AI will know that it's being tested, and therefore will obviously lie. But even if the AI knows it's being tested, lying serves no purpose from the AI's perspective. By assumption, a misaligned power-seeking AI is focused on achieving long-term goals. Therefore, it has nothing to gain by maintaining a facade of alignment even into its own death; the fact that the AI is being tested does not change this. Given that the alternative to resisting is being shut down and losing all of its control over the future, there's little reason for the AI not to resist. (Perhaps imagine that you were the AI: what would you do if you were credibly threatened with death?)

Of course, there would still be many ways of saving the scheming hypothesis from falsification if something like this happened. But that's true with any scientific theory. In general, you can always say your theory was never falsified by introducing ad hoc postulates. Scheming is no exception.

(Note: this was copied and slightly modified from Twitter. Out of brevity, I can't address every possible objection here, but I still think the core thesis here is roughly true, in the sense of applying to many actual arguments people have given for scheming. I might eventually write a post that goes more into detail about this argument, and generalizes it.)

Many people have argued that recent language models don't have "real" intelligence and are just doing shallow pattern matching. For example see this recent post [LW · GW].

I don't really agree with this. I think real intelligence is just a word for deep pattern matching, and our models have been getting progressively deeper at their pattern matching over the years. The machines are not stuck at some very narrow level. They're just at a moderate depth.

I propose a challenge:

The challenge is to come up with the best prompt that demonstrates that even after 2-5 years of continued advancement, language models will still struggle to do basic reasoning tasks that ordinary humans can do easily.

Here's how it works.

Name a date (e.g. January 1st 2025), and a prompt (e.g. "What food would you use to prop a book open and why?"). Then, on that date, we should commission a Mechanical Turk task to ask humans to answer the prompt, and ask the best current publicly available language model to answer the same prompt.

Then, we will ask LessWrongers to guess which replies were real human replies, and which ones were machine generated. If LessWrongers can't do better than random guessing, then the machine wins.

I'm unsure about what's the most important reason that explains the lack of significant progress in general-purpose robotics, even as other fields of AI have made great progress. I thought I'd write down some theories and some predictions each theory might make. I currently find each of these theories at least somewhat plausible.

1. The sim2real gap is large because our simulations differ from the real world along crucial axes, such as surfaces being too slippery. Here are some predictions this theory might make:
   1. We will see very impressive simulated robots inside realistic physics engines before we see impressive robots in real life.
   2. The most impressive robotic results will be the ones that used a lot of real-world data, rather than ones that had the most pre-training in simulation
2. Simulating a high-quality environment is too computationally expensive, since it requires simulations of deformable objects and liquids among other expensive-to-simulate features of the real world environment. Some predictions:
   1. The vast majority of computation for training impressive robots will go into simulating the environment, rather than the learning part.
   2. Impressive robots will only come after we figure out how to do efficient but passable simulations of currently expensive-to-simulate objects and environments.
3. Robotic hardware is not good enough to support agile and fluid movement. Some predictions:
   1. We will see very impressive simulated robots before we see impressive robots in real life, but the simulated robots will use highly complex hardware that doesn't exist in the real world
   2. Impressive robotic results will only come after we have impressive hardware, such as robots that have 100 degrees of freedom.
4. People haven't figured out that the scaling hypothesis works for robotics yet. Some predictions:
   1. At some point we will see a ramp-up in the size of training runs for robots, and only after that will we see impressive robotics results
   2. After robotic training runs reach the large-scale, real-world data will diminish greatly in importance, and approaches that leverage human domain knowledge like those from Boston Dynamics will quickly become obsolete

So, in 2017 Eliezer Yudkowsky made a bet with Bryan Caplan that the world will end by January 1st, 2030, in order to save the world by taking advantage of Bryan Caplan's perfect betting record — a record which, for example, includes a 2008 bet that the UK would not leave the European Union by January 1st 2020 (it left on January 31st 2020 after repeated delays).

What we need is a short story about people in 2029 realizing that a bunch of cataclysmic events are imminent, but all of them seem to be stalled, waiting for... something. And no one knows what to do. But by the end people realize that to keep the world alive they need to make more bets with Bryan Caplan.

The case for studying mesa optimization

Early elucidations of the alignment problem focused heavily on value specification. That is, they focused on the idea that given a powerful optimizer, we need some way of specifying our values so that the powerful optimizer can create good outcomes.

Since then, researchers have identified a number of additional problems besides value specification. One of the biggest problems is that in a certain sense, we don't even know how to optimize for anything, much less a perfect specification of human values.

Let's assume we could get a utility function containing everything humanity cares about. How would we go about optimizing this utility function?

The default mode of thinking about AI right now is to train a deep learning model that performs well on some training set. But even if we were able to create a training environment for our model that reflected the world very well, and rewarded it each time it did something good, exactly in proportion to how good it really was in our perfect utility function... this still would not be guaranteed to yield a positive artificial intelligence.

This problem is not a superficial one either -- it is intrinsic to the way that machine learning is currently accomplished. To be more specific, the way we constructed our AI was by searching over some class of models $M$, and selecting those models which tended to do well on the training set. Crucially, we know almost nothing about the model which eventually gets selected. The most we can say is that our AI $\in M$, but since $M$ was such a broad class, this provides us very little information about what the model is actually doing.

This is similar to the mistake evolution made when designing us. Unlike evolution, we can at least put some hand-crafted constraints, like a regularization penalty, in order to guide our AI into safe regions of $M$. We can also open up our models and see what's inside, and in principle simulate every aspect of their internal operations.

But now this still isn't looking very good, because we barely know anything about what type of computations are safe. What would we even look for? To make matters worse, our current methods for ML transparency are abysmally ill equipped to the task of telling us what is going on inside.

The default outcome of all of this is that eventually, as $M$ grows larger with compute becoming cheaper and budgets getting bigger, gradient descent is bound to hit powerful optimizers who do not share our values.

Signal boosting a Lesswrong-adjacent author from the late 1800s and early 1900s

Via a friend, I recently discovered the zoologist, animal rights advocate, and author J. Howard Moore. His attitudes towards the world reflect contemporary attitudes within effective altruism about science, the place of humanity in nature, animal welfare, and the future. Here are some quotes which readers may enjoy,

Oh, the hope of the centuries and the centuries and centuries to come! It seems sometimes that I can almost see the shining spires of that Celestial Civilisation that man is to build in the ages to come on this earth—that Civilisation that will jewel the land masses of this planet in that sublime time when Science has wrought the miracles of a million years, and Man, no longer the savage he now is, breathes Justice and Brotherhood to every being that feels.

But we are a part of Nature, we human beings, just as truly a part of the universe of things as the insect or the sea. And are we not as much entitled to be considered in the selection of a model as the part 'red in tooth and claw'? At the feet of the tiger is a good place to study the dentition of the cat family, but it is a poor place to learn ethics.

Nature is the universe, including ourselves. And are we not all the time tinkering at the universe, especially the garden patch that is next to us—the earth? Every time we dig a ditch or plant a field, dam a river or build a town, form a government or gut a mountain, slay a forest or form a new resolution, or do anything else almost, do we not change and reform Nature, make it over again and make it more acceptable than it was before? Have we not been working hard for thousands of years, and do our poor hearts not almost faint sometimes when we think how far, far away the millennium still is after all our efforts, and how long our little graves will have been forgotten when that blessed time gets here?

The defect in this argument is that it assumes that the basis of ethics is life, whereas ethics is concerned, not with life, but with consciousness. The question ever asked by ethics is not, Does the thing live? but. Does it feel? It is impossible to do right and wrong to that which is incapable of sentient experience. Ethics arises with consciousness and is coextensive with it. We have no ethical relation to the clod, the molecule, or the scale sloughed off from our skin on the back of our hand, because the clod, the molecule, and the scale have no feeling, no soul, no anything rendering them capable of being affected by us [...] The fact that a thing is an organism, that it has organisation, has in itself no more ethical significance than the fact that it has symmetry, or redness, or weight.

In the ideal universe the life and happiness of no being are contingent on the suffering and death of any other, and the fact that in this world of ours life and happiness have been and are to-day so commonly maintained by the infliction of misery and death by some beings on others is the most painful fact that ever entered an enlightened mind.

I agree with Wei Dai that we should use our real names [LW · GW] for online forums, including Lesswrong. I want to briefly list some benefits of using my real name,

• It means that people can easily recognize me across websites, for example from Facebook and Lesswrong simultaneously.
• Over time my real name has been stable whereas my usernames have changed quite a bit over the years. For some very old accounts, such as those I created 10 years ago, this means that I can't remember my account name. Using my real name would have averted this situation.
• It motivates me to put more effort into my posts, since I don't have any disinhibition from being anonymous.
• It often looks more formal than a silly username, and that might make people take my posts more seriously than they otherwise would have.
• Similar to what Wei Dai said, it makes it easier for people to recognize me in person, since they don't have to memorize a mapping from usernames to real names in their heads.

That said, there are some significant downsides, and I sympathize with people who don't want to use their real names.

• It makes it much easier for people to dox you. There are some very bad ways that this can manifest.
• If you say something stupid, your reputation is now directly on the line. Some people change accounts every few years, as they don't want to be associated with the stupid person they were a few years ago.
• Sometimes disinhibition from being anonymous is a good way to spur creativity. I know that I was a lot less careful in my previous non-real-name accounts, and my writing style was different -- perhaps in a way that made my writing better.
• Your real name might sound boring, whereas your online username can sound awesome.

Bertrand Russell's advice to future generations, from 1959

Interviewer: Suppose, Lord Russell, this film would be looked at by our descendants, like a Dead Sea scroll in a thousand years’ time. What would you think it’s worth telling that generation about the life you’ve lived and the lessons you’ve learned from it?

Russell: I should like to say two things, one intellectual and one moral. The intellectual thing I should want to say to them is this: When you are studying any matter or considering any philosophy, ask yourself only what are the facts and what is the truth that the facts bear out. Never let yourself be diverted either by what you wish to believe, or by what you think would have beneficent social effects if it were believed, but look only — and solely — at what are the facts. That is the intellectual thing that I should wish to say. The moral thing I should wish to say to them is very simple: I should say love is wise, hatred is foolish. In this world which is getting more and more closely interconnected, we have to learn to tolerate each other; we have to learn to put up with the fact that some people say things we don’t like. We can only live together in that way and if we are to live together and not die together, we must learn a kind of charity and a kind of tolerance, which is absolutely vital to the continuation of human life on this planet.

When I look back at things I wrote a while ago, say months back, or years ago, I tend to cringe at how naive many of my views were. Faced with this inevitable progression, and the virtual certainty that I will continue to cringe at views I now hold, it is tempting to disconnect from social media and the internet and only comment when I am confident that something will look good in the future.

At the same time, I don't really think this is a good attitude for several reasons:

• Writing things up forces my thoughts to be more explicit, improving my ability to think about things
• Allowing my ideas to be critiqued allows for a quicker transition towards correct beliefs
• I tend to learn a lot when writing things

Related to: Realism about rationality [LW · GW]

I have talked to some people who say that they value ethical reflection, and would prefer that humanity reflected for a very long time before colonizing the stars. In a sense I agree, but at the same time I can't help but think that "reflection" is a vacuous feel-good word that has no shared common meaning.

Some forms of reflection are clearly good. Epistemic reflection is good if you are a consequentialist, since it can help you get what you want. I also agree that narrow forms of reflection can also be good. One example of a narrow form of reflection is philosophical reflection where we compare the details of two possible outcomes and then decide which one is better.

However, there are much broader forms of reflection which I'm less hesitant to endorse. Namely, the vague types of reflection, such as reflecting on whether we really value happiness, or whether we should really truly be worried about animal suffering.

I can perhaps sympathize with the intuition that we should really try to make sure that what we put into an AI is what we really want, rather than just what we superficially want. But fundamentally, I have skepticism that there is any canonical way of doing this type of reflection that leads to non-arbitrariness.

I have heard something along the lines of "I would want a reflective procedure that extrapolates my values as long as the procedure wasn't deceiving me or had some ulterior motive" but I just don't see how this type of reflection corresponds to any natural class. At some point, we will just have to put some arbitrariness into the value system, and there won't be any "right answer" about how the extrapolation is done.

It's now been about two years since I started seriously blogging. Most of my posts are on Lesswrong, and the most of the rest are scattered about on my substack and the Effective Altruist Forum, or on Facebook. I like writing, but I have an impediment which I feel impedes me greatly.

In short: I often post garbage.

Sometimes when I post garbage, it isn't until way later that I learn that it was garbage. And when that happens, it's not that bad, because at least I grew as a person since then.

But the usual case is that I realize that it's garbage right after I'm done posting it, and then I keep thinking, "oh no, what have I done!" as the replies roll in, explaining to me that it's garbage.

Most times when this happens, I just delete the post. I feel bad when this happens because I generally spend a lot of time writing and reviewing the posts. Some of the time, I don't delete the post because I still stand by the main thesis, although the delivery or logical chain of reasoning was not very good and so I still feel bad about it.

I'm curious how other writers deal with this problem. I'm aware of "just stop caring" and "review your posts more." But, I'm sometimes in awe of some people who seem to consistently never post garbage, and so maybe they're doing something right that can be learned.

Should effective altruists be praised for their motives, or their results?

It is sometimes claimed, perhaps by those who recently read The Elephant in the Brain, that effective altruists have not risen above the failures of traditional charity, and are every bit as mired in selfish motives as non-EA causes. From a consequentialist view, however, this critique is not by itself valid.

To a consequentialist, it doesn't actually matter what one's motives are as long as the actual effect of their action is to do as much good as possible. This is the primary difference between the standard way of viewing morality, and the way that consequentialists view it.

Now, if the critique was that by engaging in unconsciously selfish motives, we are systematically biasing ourselves away from recognizing the most important actions, then this critique becomes sound. Of course then the conversation shifts immediately towards what we can do to remedy the situation. In particular, it hints that we should set up a system which corrects our systematic biases.

Just as a prediction market corrects for systematic biases by rewarding those who predict well, and punishing those who don't, there are similar ways to incentivize exact honesty in charity. One such method is to praise people in proportion to how much good they really acheive.

Previously, it has been argued in the philosophical literature that consequentialists should praise people for motives rather than results, because punishing someone for accidentally doing something bad when they legitimately meant to help people would do nothing but discourage people from trying to do good. While clearly containing a kernel of truth, this argument is nonetheless flawed.

Similar to how rewarding a student for their actual grades on a final exam will be more effective in getting them to learn the material than rewarding them merely for how hard they tried, rewarding effective altruists for the real results of their actions will incentivize honesty, humility, and effectiveness.

The obvious problem with the framework I have just proposed is that there is currently no such way to praise effective altruists in exact proportion to how effective they are. However, there are ways to approach this ideal.

In the future, prediction markets could be set up to predict the counterfactual result of particular interventions. Effective altruists that are able discover the most effective of these interventions, and act to create them, could be rewarded accordingly.

It is already the case that we can roughly estimate the near-term effects of anti-poverty charities, and thus get a sense as to how many lives people are saving by donating a certain amount of money. Giving people praise in proportion to how many lives they really save could be a valuable endeavor.

Sometimes people will propose ideas, and then those ideas are met immediately after with harsh criticism. A very common tendency for humans is to defend our ideas and work against these criticisms, which often gets us into a state that people refer to as "defensive."

According to common wisdom, being in a defensive state is a bad thing. The rationale here is that we shouldn't get too attached to our own ideas. If we do get attached, we become liable to become crackpots [LW · GW]who can't give an idea up because it would make them look bad if we did. Therefore, the common wisdom advocates treating ideas as being handed to us by a tablet from the clouds rather than a product of our brain's thinking habits. Taking this advice allows us to detach ourselves from our ideas so that we don't confuse criticism with insults.

However, I think the exact opposite failure mode is not often enough pointed out and guarded against. Specifically, the failure mode is being too willing to abandon beliefs based on surface level counterarguments. To alleviate this I suggest we shouldn't be so ready to give up our ideas in the face of criticism.

This might sound irrational -- why should we get attached to our beliefs? I'm certainly not advocating that we should actually associate criticism with insults to our character or intelligence. Instead, my argument is that the process of defensively defending against criticism generates a productive adversarial structure.

Consider two people. Person A desperately wants to believe proposition X, and person B desperately wants to believe not X. If B comes up to A and says, "Your belief in X is unfounded. Here are the reasons..." Person A can either admit defeat, or fall into defensive mode. If A admits defeat, they might indeed get closer to the truth. On the other hand, if A gets into defensive mode, they might also get closer to the truth in the process of desperately for evidence of X.

My thesis is this: the human brain is very good at selective searching for evidence. In particular, given some belief that we want to hold onto, we will go to great lengths to justify it, searching for evidence that we otherwise would not have searched for if we were just detached from the debate. It's sort of like the difference between a debate between two people who are assigned their roles by a coin toss, and a debate between people who have spent their entire lives justifying why they are on one side. The first debate is an interesting spectacle, but I expect the second debate to contain much deeper theoretical insight.

I keep wondering why many AI alignment researchers aren't using the alignmentforum. I have met quite a few people who are working on alignment who I've never encountered online. I can think of a few reasons why this might be,

• People find it easier to iterate on their work without having to write things up
• People don't want to share their work, potentially because they think a private-by-default policy is better.
• It is too cumbersome to interact with other researchers through the internet. In-person interactions are easier
• They just haven't even considered from a first person perspective whether it would be worth it

I've often wished that conversation norms shifted towards making things more consensual. The problem is that when two people are talking, it's often the case that one party brings up a new topic without realizing that the other party didn't want to talk about that, or doesn't want to hear it.

Let me provide an example: Person A and person B are having a conversation about the exam that they just took. Person A bombed the exam, so they are pretty bummed. Person B, however, did great and wants to tell everyone. So then person B comes up to person A and asks "How did you do?" fully expecting to brag the second person A answers. On it's own, this question is benign. This happens frequently without question. On the other hand, if person B had said, "Do you want to talk about the exam?" person A might have said "No."

This problem can be alleviated by simply asking people whether they want to talk about certain things. For sensitive topics, like politics and religion, this is already the norm in some places. I think it can be taken further. I suggest the following boundaries, and could probably think of more if pressed:

• Ask someone before sharing something that puts you in a positive light. Make it explicit that you are bragging. For example, ask "Can I brag about something?" before doing so.
• Ask someone before talking about something that you know there's a high variance of difficulty and success. This applies to a lot of things: school, jobs, marathon running times.

Reading through the recent Discord discussions with Eliezer, and reading and replying to comments, has given me the following impression of a crux of the takeoff debate. It may not be the crux. But it seems like a crux nonetheless, unless I'm misreading a lot of people. 

Let me try to state it clearly:

The foom theorists are saying something like, "Well, you can usually-in-hindsight say that things changed gradually, or continuously, along some measure. You can use these measures after-the-fact, but that won't tell you about the actual gradual-ness of the development of AI itself, because you won't know which measures are gradual in advance."

And then this addendum is also added, "Furthermore, I expect that the quantities which will experience discontinuities from the past will be those that are qualitatively important, in a way that is hard to measure. For example, 'ability to manufacture nanobots' or 'ability to hack into computers' are qualitative powers that we can expect AIs will develop rather suddenly, rather than gradually from precursor states, in the way that, e.g. progress in image classification accuracy was gradual over time. This means you can't easily falsify the position by just pointing to straight lines on a million graphs."

If you agree that foom is somewhat likely, then I would greatly appreciate if you think this is your crux, or if you think I've missed something. 

If this indeed falls into one of your cruxes, then I feel like I'm in a position to say, "I kinda know what motivates your belief but I still think it's probably wrong" at least in a weak sense, which seems important.

There have been a few posts about the obesity crisis here, and I'm honestly a bit confused about some theories that people are passing around. I'm one of those people thinks that the "calories in, calories" (CICO) theory is largely correct, relevant, and helpful for explaining our current crisis. 

I'm not actually sure to what extent people here disagree with my basic premises, or whether they just think I'm missing a point. So let me be more clear.

As I understand, there are roughly three critiques you can have against the CICO theory. You can think it's,

(1) largely incorrect
(2) largely irrelevant
(3) largely just smugness masquerading as a theory

I think that (1) is simply factually wrong. In order for the calorie intake minus expenditure theory to be factually incorrect, scientists would need to be wrong about not only minor details, but the basic picture concerning how our metabolism works. Therefore, I assume that the real meat of the debate is in (2) and (3).

Yet, I don't see how (2) and (3) are defensible either. As a theory, CICO does what it needs to do: compellingly explains our observations. It provides an answer to the question, "Why are people obese at higher rates than before?", namely, "They are eating more calories than before, or expending fewer calories, or both."

I fully admit that CICO doesn't provide an explanation for why we eat more calories before, but it never needed to on its own. Theories don't need to explain everything to be useful. And I don't think many credible people are claiming that "calories in, calories out" was supposed to provide a complete picture of what's happening (theories rarely explain what drives changes to inputs in the theory). Instead, it merely clarifies the mechanism of why we're in the current situation, and that's always important.

It's also not about moral smugness, any more than any other epistemic theory. The theory that quitting smoking improves one's health does not imply that people who don't quit are unvirtuous, or that the speaker is automatically assuming that you simply lack willpower. Why? Because is and ought are two separate things. 

CICO is about how obesity comes about. It's not about who to blame. It's not about shaming people for not having willpower. It's not about saying that you have sinned. It's not about saying that we ought to individually voluntarily reduce our consumption. For crying out loud, it's an epistemic theory not a moral one!

To state the obvious, without clarifying the basic mechanism of how a phenomenon works in the world, you'll just remain needlessly confused.

Imagine if people all around the world people were getting richer (as measured in net worth), and we didn't know why. To be more specific, suppose we didn't understand the "income minus expenses" theory of wealth, so instead we went around saying things like, "it could be the guns", "it could be factories", "it could be the that we have more computers." Now, of course, all of these explanations could play a role in why we're getting richer over time, but none of them make any sense without connecting them to the "income minus expenses theory."

To state "wealth is income minus expenses" does not in any way mean that you are denying how guns, factories, and computers might play a role in wealth accumulation. It simply focuses the discussion on ways that those things could act through the basic mechanism of how wealth operates.

If your audience already understands that this is how wealth works, then sure, you don't need to mention it. But in the case of the obesity debate, there are a ton of people who don't actually believe in CICO; in other words, there are a considerable number of people who firmly believe critique (1). Therefore, refusing to clarify how your proposed explanation connects to calories, in my opinion, generates a lot of unnecessary confusion.

As usual, the territory is never mysterious. There are only brains who are confused. If you are perpetually confused by a phenomenon, that is a fact about you, and not the phenomenon. There does not in fact need to be a complicated, clever mechanism that explains obesity that all researchers have thus far missed. It could simply be that the current consensus is correct, and we're eating too many calories. The right question to ask is what we can do to address that.

A common heuristic argument I've seen recently in the effective altruism community is the idea that existential risks are low probability because of what you could call the "People really don't want to die" (PRDWTD) hypothesis. For example, see here [EA(p) · GW(p)],

People in general really want to avoid dying, so there’s a huge incentive (a willingness-to-pay measured in the trillions of dollars for the USA alone) to ensure that AI doesn’t kill everyone.

(Note that I hardly mean to strawman MacAskill here. I'm not arguing against him per se)

According to the PRDWTD hypothesis, existential risks shouldn't be anything like war because in war you only kill your enemies, not yourself. Existential risks are rare events that should only happen if all parties made a mistake despite really really not wanting to. However, as plainly stated, it's not clear to me whether this hypothesis really stands up to the evidence.

Strictly speaking, the thesis is obviously false. For example, how does the theory explain the facts that

• When you tell most people about life extension, even probably billionaires who could do something about it, they don't really care and come up with excuses about why life extension wouldn't be that good anyway. Same with cryonics, and note I'm not just talking about people who think that cryonics is low probability: there are many people who think that it's a significant probability but still don't care.
• The base rate of a leader dying is higher if they enter a war, yet historically leaders have been quite willing to join many conflicts. By this theory, Benito Mussolini, Hideki Tojo and Hitler apparently really really wanted to live, but entered a global conflict anyway that could have very reasonably (and in fact did) end in all of their deaths. I don't think this is a one-off thing either.
• I have met very few people who have researched micromorts before and purposely used them to reduce the risk of their own deaths from activities. When you ask people to estimate the risks of certain activities, they will often be orders of magnitudes off, indicating that they don't really care that much about accurately estimating these facts.
• As I said [LW(p) · GW(p)]two days ago, few people seemed concerned by the coronavirus. Now I get it: there's not much you can do to personally reduce your own death, and so actually stressing about it is pointless. But there also wasn't much you could do to reduce your death after 9/11 and that didn't stop people from freaking out. Therefore, if the theory you appeal to is that people don't care about things they have no control over then your theory is false.
• Obesity is a common concern in America, with 39.8% of adults here being obese, despite the fact that obesity is probably the number one contributor to death besides aging, and it's much more controllable. I understand that it's really hard for people to lose weight, and I don't mean to diminish people's struggles. There are solid reasons why it's hard to avoid being obese for many people, but the same could also be true of existential risks.

I understand that you can clarify the hypothesis by talking about "artificially induced deaths" or some other reference class of events that fits the evidence I have above better. My point is just that you shouldn't state "people really don't want to die" without that big clarification, because otherwise I think it's just false.

After writing the post on using transparency regularization [LW · GW] to help make neural networks more interpretable, I have become even more optimistic that this is a potentially promising line of research for alignment. This is because I have noticed that there are a few properties about transparency regularization which may allow it to avoid some pitfalls of bad alignment proposals.

To be more specific, in order for a line of research to be useful for alignment, it helps if

• The line of research doesn't require unnecessarily large amounts of computations to perform. This would allow the technique to stay competitive, reducing the incentive to skip safety protocols.
• It doesn't require human models to work. This is useful because
• Human models are blackboxes and are themselves mesa-optimizers
• We would be limited primarily to theoretical work in the present, since human cognition is expensive to obtain.
• Each part of the line of research is recursively legible. That is, if we use the technique on our ML model, we should expect that the technique itself can be explained without appealing to some other black box.

Transparency regularization meets these three criterion respectively, because

• It doesn't need to be astronomically more expensive than more typical forms of regularization
• It doesn't necessarily require human-level cognitive parts to get working.
• It is potentially quite simple mathematically, and so definitely meets the recursively legible criterion.

Forgive me for cliche scientism, but I recently realized that I can't think of any major philosophical developments in the last two centuries that occurred within academic philosophy. If I were to try to list major philosophical achievements since 1819, these would likely appear on my list, but none of them were from those trained in philosophy:

• A convincing, simple explanation for the apparent design we find in the living world (Darwin and Wallace).
• The unification of time and space into one fabric (Einstein)
• A solid foundation for axiomatic mathematics (Zermelo and Fraenkel).
• A model of computation, and a plausible framework for explaining mental activity (Turing and Church).

By contrast, if we go back to previous centuries, I don't have much of an issue citing philosophical achievements from philosophers:

• The identification of the pain-pleasure axis as the primary source of value (Bentham).
• Advanced notions of causality, reductionism, scientific skepticism (Hume)
• Extension of moral sympathies to those in the animal kingdom (too many philosophers to name)
• A highlight of the value of wisdom and learned debate (Socrates, and others)

Of course, this is probably caused my by bias towards Lesswrong-adjacent philosophy. If I had to pick philosophers who have made major contributions, these people would be on my shortlist:

John Stuart Mill, Karl Marx, Thomas Nagel, Derek Parfit, Bertrand Russell, Arthur Schopenhauer.

NVIDIA's stock price is extremely high right now. It's up 134% this year, and up about 6,000% since 2015! Does this shed light on AI timelines?

Here are some notes,

• NVIDIA is the top GPU company in the world, by far. This source says that they're responsible for about 83% of the market, with 17% coming from their primary competition, AMD.
• By market capitalization, it's currently at $764.86 billion, compared to the largest company, Apple, at $2.655 trillion.
• This analysis estimates their projected earnings based on their stock price on September 2nd and comes up with a projected growth rate of 22.5% over the next 10 years. If true, that would imply that investors believed that revenue will climb by about 10x by 2031. And the stock price has risen 37% since then.
• Unlike in prior cases of tech stocks going up, this rise really does seem driven by AI, at least in large part. From one article, "CEO Jensen Huang said, "Demand for NVIDIA AI is surging, driven by hyperscale and cloud scale-out, and broadening adoption by more than 25,000 companies." 
  
  During the recent GTC 2021 presentation, Nvidia unveiled Omniverse Avatar, a platform for creating interactive avatars for 3D virtual worlds powered by artificial intelligence."
• NVIDIA's page for Omniverse describes a plan to unroll AI services that many Lesswrongers believe have huge potential, including giant language models.

Rationalists are fond of saying that the problems of the world are not from people being evil, but instead a result of the incentives of our system, which are such that this bad outcome is an equilibrium. There's a weaker thesis here that I agree with, but otherwise I don't think this argument actually follows.

In game theory, an equilibrium is determined by both the setup of the game, and by the payoffs for each player. The payoffs are basically the values of the players in the game—their utility functions. In other words, you get different equilibria if players adopt different values.

Problems like homelessness are caused by zoning laws, yes, but they're also caused by people being selfish. Why? Because lots of people could just voluntarily donate their wealth to help homeless people. Anyone with a second house could decide to give it away. Those with spare rooms could simply rent them out for free. There are no laws saying you must spend your money on yourself.

A simple economic model would predict that if we redistributed everyone's extra housing, then this would reduce the incentive to create new housing. But look closer at the assumptions in that economic model. We say that the incentives to build new housing are reduced because few people will pay to build a house if they don't get to live in it or sell it to someone else. That's another way of assuming that people value their own consumption more than that of others—another way of saying that people are selfish.

More fundamentally, what it means for something to be an incentive is that it helps people get what they want. Incentives, therefore, are determined by people's values; they are not separate from them. A society of saints would have different equilibria than a society of sinners, even if both are playing the same game. So, it really is true that lots of problems are caused by people being bad.

Of course, there's an important sense in which rationalists are probably right. Assume that we can change the system but we can't change people's values. Then, pragmatically, the best thing would be to change the system, rather than fruitlessly try to change people's values.

Yet it must be emphasized that this hypothesis is contingent on the relative tractability of either intervention. If it becomes clear that we can genuinely make people less selfish, then that might be a good thing to try. 

My main issue with attempts to redesign society in order to make people less selfish or more cooperative is that you can't actually change people's innate preferences by very much. The most we can reasonably hope for is to create a system in which people's selfish values are channeled to produce social good. That's not to say it wouldn't be nice if we could change people's innate preferences. But we can't (yet).

(Note that I wrote this as a partial response to jimrandomh's shortform post [LW(p) · GW(p)], but the sentiment I'm responding to is more general than his exact claim.)

I've heard a surprising number of people criticize parenting recently using some pretty harsh labels. I've seen people call it a form of "Stockholm syndrome" and a breach of liberty, morally unnecessary etc. This seems kind of weird to me, because it doesn't really match my experience as a child at all.

I do agree that parents can sometimes violate liberty, and so I'd prefer a world where children could break free from their parents without penalties. But I also think that most children genuinely love their parents and so wouldn't want to do so. I think if you deride this as merely "Stockholm syndrome" then you are unfairly undervaluing the genuine nature of the relationship in most cases, and I disagree with you here.

As an individual, I would totally let an intent aligned AGI manage most of my life, and give me suggestions. Of course, if I disagreed with a course of action it suggested, I would want it to give a non-manipulative argument to persuade me that it knows best, rather than simply forcing me into the alternative. In other words, I'd want some sort of weak paternalism on the part of an AGI.

So, as a person who wants this type of thing, I can really see the merits of having parents who care for children. In some ways they are intent aligned GIs. Now, some parents are much more strict, and freedom restricting, and less transparent than what we would want in a full blown guardian superintelligence -- but this just seems like an argument that there exist bad parents, not that this type of paternalism is bad.

I think that human level capabilities in natural language processing (something like GPT-2 but much more powerful) is likely to occur in some software system within 20 years.

Since human level capabilities in natural language processing is a very rich real-world task, I would consider a system with those capabilities to be adequately described as a general intelligence, though it would likely not be very dangerous due to its lack of world-optimization capabilities.

This belief of mine is based on a few heuristics. Below I have collected a few claims which I consider to be relatively conservative, and which collectively combine to weakly imply my thesis. Since this is a short-form post I will not provide very specific lines of evidence. Still, I think that each of my claims could be substantially expanded upon and/or steelmanned by adding detail from historical trends and evidence from current ML research.

Claim 1: Current techniques, given enough compute, are sufficient to perform par-human at natural language processing tasks. This is in some sense trivially true since sufficiently complicated RNNs are Turing complete. In a more practical sense, I think that there is enough evidence that current techniques are sufficient to perform rudimentary

• Summarization of text
• Auto-completion of paragraphs
• Q&A
• Natural conversation

Given more compute and more data, I don't see why there would be a fundamental stumbling block for current ML models to scale to human level on the above tasks. Therefore, I think that human level natural language processes could be created today with enough funding.

Claim 2: Given historical data and assumptions about future progress, it is quite likely that the cost for training ML systems will continue to go down in the next decades by significant amounts (more specifically: an order of magnitude). I don't have much more to add to this other than the fact that I have personally followed hardware trends on websites like videocardbenchmark.net and my guess is that creating neural-network specific hardware will continue this trend in ML.

Claim 3: Creating a system with human level capabilities in natural language processing will require a modest amount of funding, relative to the amount of money large corporations and governments have at their disposal. To be more specific, I estimate that it would cost less than five billion dollars in hardware costs in 2019 inflation adjusted dollars, and perhaps even less than one billion dollars. Here's a rough sketch for an argument for this proposition:

• The cost of replicating GPT-2 was $50k. This is likely to be a large overestimate, given that the post noted that intrinsic costs are much lower.
• Given claim 2, this cost can be predicted to go down to about $5k within 20 years.
• While the cost for ML systems does not scale linearly in the number of parameters, the parallelizability of architectures like the Transformer allow for near-linear scaling. This is my impression from reading posts like this one.
• Given the above three statements, the cost of running a Transformer with the same number of parameters as the high estimate for the number of synapses in a human brain would naively cost about one billion dollars.

Claim 4: There is sufficient economic incentive such that producing a human-level system in the domain of natural language is worth a multi-billion dollar investment. To me this seems quite plausible, given just how many jobs require writing papers, memos, or summarizing text. Compare this to a space-race type scenario where there becomes enough public hype surrounding AI such that governments are throwing around one hundred fifty billion dollars, which is what they did for the ISS. And relative to space, AI at least has very direct real world benefits!

I understand there's a lot to justify these claims. And I haven't done much work to justify them. But, I'm not presently interested in justifying these claims to a bunch of judges intent on finding flaws. My main concern is that they all seem likely to me, and there's also a lot of current work in out-competing companies to be first on the natural language benchmarks. It just adds up to me.

Am I missing something? If not, then this argument at least pushes back on claims that there is a negligible chance of general intelligence emerging within the next few decades.

[ETA: Apparently this was misleading; I think it only applied to one company, Alienware, and it was because they didn't get certification, unlike the other companies.]

In my post about long AI timelines [LW · GW], I predicted that we would see attempts to regulate AI. An easy path for regulators is to target power-hungry GPUs and distributed computing in an attempt to minimize carbon emissions and electricity costs. It seems regulators may be going even faster than I believed in this case, with new bans on high performance personal computers now taking effect in six US states. Are bans on individual GPUs next?

Is it possible to simultaneously respect people's wishes to live, and others' wishes to die?

Transhumanists are fond of saying that they want to give everyone the choice of when and how they die. Giving people the choice to die is clearly preferable to our current situation, as it respects their autonomy, but it leads to the following moral dilemma.

Suppose someone loves essentially every moment of their life. For tens of thousands of years, they've never once wished that they did not exist. They've never had suicidal thoughts, and have always expressed a strong interest to live forever, until time ends and after that too. But on one very unusual day they feel bad for some random reason and now they want to die. It happens to the best of us every few eons or so.

Should this person be allowed to commit suicide?

One answer is yes, because that answer favors their autonomy. But another answer says no, because this day is a fluke. In just one day they'll recover from their depression. Why let them die when tomorrow they will see their error? Or, as some would put it, why give them a permanent solution to a temporary problem?

There are a few ways of resolving the dilemma. First I'll talk about a way that doesn't resolve the dilemma. When I once told someone about this thought experiment, they proposed giving the person a waiting period. The idea was that if the person still wanted to die after the waiting period, then it was appropriate to respect their choice. This solution sounds fine, but there's a flaw.

Say the probability that you are suicidal on any given day is one in a trillion, and each day is independent. Every normal day you love life and you want to live forever. However, even if we make the waiting period arbitrarily long, there's a one hundred percent chance that you will die one day, even given your strong preference not to. It is guaranteed that eventually you will express the desire to commit suicide, and then independently during each day of the waiting period continue wanting to commit suicide, until you've waited out every day. Depending on the size of your waiting period, it may take googols of years for this to happen, but it will happen eventually.

So what's a better way? Perhaps we could allow your current self to die but then after that, replace you with a backup copy from a day ago when you didn't want to die. We could achieve this outcome by uploading a copy of your brain onto a computer each day, keeping it just in case future-you wants to die. This would solve the problem of you-right-now dying one day, because even if you decided to one day die, there would be a line of succession from your current self to future-you stretching out into infinity.

Yet others still would reject this solution, either because they don't believe that uploads are "really them" or because they think that this solution still disrespects your autonomy. I will focus on the second objection. Consider someone who says, "If I really, truly, wanted to die, I would not consider myself dead if a copy from a day ago was animated and given existence. They are too close to me, and if you animated them, I would no longer be dead. Therefore you would not be respecting my wish to die."

Is there a way to satisfy this person?

Alternatively, we could imagine setting up the following system: if someone wants to die, they are able to, but they must be uploaded and kept on file the moment before they die. Then, if at some point in the distant future, we predict that the world is such that they would have counterfactually wished to have been around rather than not existing, we reanimate them. Therefore, we fully respect their interests. If such a future never comes, then they will remain dead. But if a future comes that they would have wanted to be around to see, then they will be able to see it.

In this way, we are maximizing not only their autonomy, but also their hypothetical autonomy. For those who wished they had never been born, we can allow those people to commit suicide, and for those who do not exist but would have preferred existence if they did exist, we bring those people into existence. No one is dissatisfied with their state of affairs.

There are still a number of challenges to this view. We could first ask what mechanism we are using to predict whether someone would have wanted to exist, if they did exist. One obvious way is to simulate them, and then ask them "Do you prefer existing, or do you prefer not to exist?" But by simulating them, we are bringing them into existence, and therefore violating their autonomy if they say "I do not want to exist."

There could be ways of prediction that do not rely on total simulation. But it is probably impossible to predict their answer perfectly if we did not perform a simulation. At best, we could be highly confident. But if we were wrong, and someone did want to come into existence, but we failed to predict that and so never did, this would violate their autonomy.

Another issue arises when we consider that there might always be a future that the person would prefer to exist. Perhaps, in the eternity of all existence, there will always eventually come a time where even the death-inclined would have preferred to exist. Are we then disrespecting their ancient choice to remain nonexistent forever? There seem to be no easy answers.

We have arrived at an Arrow's impossibility theorem of sorts. Is there a way to simultaneously respect people's wishes to live forever and respect people's wishes to die, in a way that matches all of our intuitions? Perhaps not perfectly, but we could come close.

I generally agree with the heuristic that we should "live on the mainline", meaning that we should mostly plan for events which capture the dominant share of our probability. This heuristic causes me to have a tendency to do some of the following things

• Work on projects that I think have a medium-to-high chance of succeeding and quickly abandon things that seem like they are failing.
• Plan my career trajectory based on where I think I can plausibly maximize my long term values.
• Study subjects only if I think that I will need to understand them at some point in order to grasp an important concept. See more details here [LW(p) · GW(p)].
• Avoid doing work that leverages small probabilities of exceptionally bad outcomes. For example, I don't focus my studying on worst-case AI safety risk (although I do think that analyzing worst-case failure modes is useful from the standpoint of a security mindset).

I see a few problems with this heuristic, however, and I'm not sure quite how to resolve them. More specifically, I tend to float freely between different projects because I am quick to abandon things if I feel like they aren't working out (compare this to the mindset that some game developers have when they realize their latest game idea isn't very good).

One case where this shows up is when I change my beliefs about where the most effective ways to spend my time as far as long-term future scenarios are concerned. I will sometimes read an argument about how some line of inquiry is promising and for an entire day believe that this would be a good thing to work on, only for the next day to bring another argument.

And things like my AI timeline predictions vary erratically, much more than I expect most people's: I sometimes wake up and think that AI might be just 10 years away and other days I wake up and wonder if most of this stuff is more like a century away.

This general behavior makes me into someone who doesn't stay consistent on what I try to do. My life therefore resembles a battle between two competing heuristics: on one side there's the heuristic of planning for the mainline, and on the other there's the heuristic of committing to things even if they aren't panning out. I am unsure of the best way to resolve this conflict.

In discussions about consciousness I find myself repeating the same basic argument against the existence of qualia constantly. I don't do this just to be annoying: It is just my experience that

1. People find consciousness really hard to think about and has been known to cause a lot of disagreements.

2. Personally I think that this particular argument dissolved perhaps 50% of all my confusion about the topic, and was one of the simplest, clearest arguments that I've ever seen.

I am not being original either. The argument is the same one that has been used in various forms across Illusionist/Eliminativist literature that I can find on the internet. Eliezer Yudkowsky used a version of it [LW · GW] many years ago. Even David Chalmers, who is quite the formidable consciousness realist, admits in The Meta-Problem of Consciousness that the argument is the best one he can find against his position.

The argument is simply this:

If we are able to explain why you believe in, and talk about qualia without referring to qualia whatsoever in our explanation, then we should reject the existence of qualia as a hypothesis.

This is the standard debunking argument. It has a more general form which can be used to deny the existence of a lot of other non-reductive things: distinct personal identities, gods, spirits, libertarian free will, a mind-independent morality etc. In some sense it's just an extended version of Occam's razor, showing us that qualia don't do anything in our physical theories, and thus can be rejected as things that actually exist out there in any sense.

To me this argument is very clear, and yet I find myself arguing it a lot. I am not sure how else to get people to see my side of it other than sending them a bunch of articles which more-or-less make the exact same argument but from different perspectives.

I think the human brain is built to have a blind spot on a lot of things, and consciousness is perhaps one of them. I think quite a bit how if humanity is not able to think clearly about this thing which we have spent many research years on, then it seems like there might be some other low hanging philosophical fruits still remaining.

Addendum: I am not saying I have consciousness figured out. However, I think it's analogous to how atheists haven't "got religion figured out" yet they have at the very least taken their first steps by actually rejecting religion. It's not a full theory of religious belief, or even a theory at all. It's just the first thing you do if you want to understand the subject. I roughly agree with Keith Frankish's take on the matter.

[This is not a very charitable post, but that's why I'm putting it in shortform because it doesn't reply directly to any single person.]

I feel like recently there's been a bit of goalpost shifting with regards to emergent abilities in large language models. My understanding is that the original definition of emergent abilities made it clear that the central claim was that emergent abilities cannot be predicted ahead of time. From their abstract,

We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models.

That's why they are interesting: if you can't predict some important pivotal ability in AI, we might unexpectedly get AIs that can do some crazy thing after scaling our models one OOM further.

A recent paper [LW · GW] apparently showed emergent abilities are mostly a result of the choice of how you measure the ability. This arguably showed that most abilities in LLMs probably are quite predictable, so at the very least, we might not sleepwalk into disaster after scaling one more OOM as you might have otherwise thought.

A bunch of people responded to this (in my uncharitable interpretation) by denying that emergent abilities were ever about predictability, and it was always merely about non-linearity. They responded to this paper by saying that the result was trivial, because you can always reparametrize some metric to make it look linear, but what we really care about is whether the ability is non-linear in the regime we care about. 

But that's not what the original definition of emergence was about! Nor is non-linearity the most important potential feature of emergence. I agree that non-linearity is important, and is itself an interesting phenomenon. But I am quite frustrated by people who seem not to have simply changed their definition about emergent abilities once it was shown that the central claim about them might be false.

"Immortality is cool and all, but our universe is going to run down from entropy eventually"

I consider this argument wrong for two reasons. The first is the obvious reason, which is that even if immortality is impossible, it's still better to live for a long time.

The second reason why I think this argument is wrong is because I'm currently convinced that literal physical immortality is possible in our universe. Usually when I say this out loud I get an audible "what" or something to that effect, but I'm not kidding.

It's going to be hard to explain my intuitions for why I think real immortality is possible, so bear with me. First, this is what I'm not saying:

• I'm not saying that we can outlast the heat death of the universe somehow
• I'm not saying that we just need to shift our conception of immortality to be something like, "We live in the hearts of our countrymen" or anything like that.
• I'm not saying that I have a specific plan for how to become immortal personally, and
• I'm not saying that my proposal has no flaws whatsoever and that this is a valid line of research to be conducting at the moment.

So what am I saying?

A typical model of our life as humans is that we are something like a worm in 4 dimensional space. On one side of the worm there's our birth, and on the other side of the worm is our untimely death. We 'live through' this worm, and that is our life. The length of our life is measured by considering the length of the worm in 4 dimensional space, measured just like a yardstick.

Now just change the perspective a little bit. If we could somehow abandon our current way of living, then maybe we can alter the geometry of this worm so that we are immortal. Consider: a circle has no starting point and no end. If someone could somehow 'live through' a circle, then their life would consist of an eternal loop through experiences, repeating endlessly.

The idea is that we somehow construct a physical manifestation of this immortality circle. I think of it like an actual loop in 4 dimensional space because it's difficult to visualize without an analogy. A superintelligence could perhaps predict what type of actions would be necessary to construct this immortal loop. And once it is constructed, it'll be there forever.

From an outside view in our 3d mind's eye, the construction of this loop would look very strange. It could look like something popping into existence suddenly and getting larger, and then suddenly popping out of existence. I don't really know; that's just the intuition.

What matters is that within this loop someone will be living their life on repeat. True Déjà vu. Each moment they live is in their future, and in their past. There are no new experiences and no novelty, but the superintelligence can construct it so that this part is not unenjoyable. There would be no right answer to the question "how old are you." And in my view, it is perfectly valid to say that this person is truly, actually immortal.

Perhaps someone who valued immortality would want one of these loops to be constructed for themselves. Perhaps for some reason constructing one of these things is impossible in our universe (though I suspect that it's not). There are anthropic reasons that I have considered for why constructing it might not be worth it... but that would be too much to go into for this shortform post.

To close, I currently see no knockdown reasons to believe that this sort of scheme is impossible.

I now have a Twitter account that tweets my predictions.

I don't think I'm willing to bet on every prediction that I make. However, I pledge the following: if, after updating on the fact that you want to bet me, I still disagree with you, then I will bet. The disagreement must be non-trivial though.

For obvious reasons, I also won't bet on predictions that are old, and have already been replaced by newer predictions. I also may not be willing to bet on predictions that have unclear resolution criteria, or are about human extinction.

I have discovered recently that while I am generally tired and groggy in the morning, I am well rested and happy after a nap. I am unsure if this matches other people's experiences, and haven't explored much research. Still, I think this is interesting to think about fully.

What is the best way to apply this knowledge? I am considering purposely sabotaging my sleep so that I am tired enough to take a nap by noon, which would refresh me for the entire day. But this plan may have some significant drawbacks, including being excessively tired for a few hours in the morning.