LessWrong 2.0 Reader

I think I'm still missing the connection. How did you get from "AI versus humanity" to "whether human immortality is desirable"?

Edit: a substantial part of my objection is to this:

If it is possible to trivially fill in the rest of his argument, then I think it is better for him to post that, instead of posting something that needs to be filled-in, and which doesn't actually back up the thesis that people are interpreting him as arguing for.

It is not worth always worth doing a three-month research project to fill in many details that you have already written up elsewhere in order to locally refute a bad argument that does not depend on those details.  (The current post does locally refute several bad arguments, including that the law of comparative advantage means it must always be more advantageous to trade with humans.  If you understand it to be making a much broader argument than that, I think that is the wrong understanding.)

Separately, it's not clear to me whether you yourself could fill in those details.  In other words, are you asking for those details to be filled in because you actually don't know how Eliezer would fill them in, or because you have some other reason for asking for that additional labor (i.e. you think it'd be better for the public discourse if all of Eliezer's essays included that level of detail)?

Original comment:

The essay is a local objection to a specific bad argument, which, yes, is more compelling if you're familiar with Eliezer's other beliefs on the subject.  Eliezer has written about those beliefs fairly extensively, and much of his writing was answering various other objections (including many of those you listed).  There does not yet exist a single ten-million-word treatise which provides an end-to-end argument of the level of detail you're looking for.  (There exist the Sequences, which are over a million words, but they while they implicitly answer many of these objections, they're not structured to be a direct argument to this effect.)

As a starting point, why does nanotech imply that it will be cheaper to disassemble humans than to trade with them?

I think it would be much cheaper for you to describe a situation where an ASI develops the kind of nanotech that'd grant it technological self-sufficiency (and the ability to kill all humans), and it remains the case that trading with humans for any longer than it takes to bootstrap that nanotech is cheaper than just doing its own thing, while still being compatible with Eliezer's model of the world.  I have no idea what kind of reasoning or justification you would find compelling as an argument for "cheaper to disassemble"; it seems to require very little additional justification conditioning on that kind of nanotech being realized.  My current guess is that you do not think that kind of nanotech is physically realizable by any ASI we are going to develop (including post-RSI), or maybe you think the ASI will be cognitively disadvantaged compared to humans in domains that it thinks are important (in ways that it can't compensate for, or develop alternatives for, somehow).

Earth as a proportion of the solar system's planetary mass is probably comparable to national parks as a proportion of the Earth's land, if not lower.

Yeah, but not if we weight that land by economic productivity, I think.

Reading your posts gives me the impression that we are both loosely pointing at the same object, but with fairly large differences in terminology and formalism. 

While computing exact counter-factuals has issues with chaos, I don't think this poses a problem for my earlier proposal. I don't think it is necessary that the AGI is able to exactly compute the counterfactual entropy production, just that it makes a reasonably accurate approximation.^[1]

I think I'm in agreement with your premise that the "constitutionalist form of agency" is flawed. IThe absence of entropy (or indeed any internal physical resource management) from the canonical Lesswrong agent foundation model is clearly a major issue. My loose thinking on this is that bayesian networks are not a natural description of the physical world at all, although they're an appropriate tool for how certain, very special types of open-systems, "agentic optimizers" model the world. 

I have had similar thoughts to what has motivated your post on the "causal backbone". I believe "the heterogenous fluctuations will sometimes lead to massive shifts in how the resources are distributed" is something we would see in a programmable, unbounded optimizer^[2]. But I'm not sure if attempting to model this as there being a "causal backbone" is the description that is going to cut reality at the joints, due to difficulties with the physicality of causality itself (see work by Jenann Ismael [LW · GW]).

1. ^{^}

   You can construct pathological environments in which the AGI's computation (with limited physical resources) of the counterfactual entropy production is arbitrarily large (and the resulting behaviour is arbitrarily bad). I don't see this as a flaw with the proposal as this issue of being able to construct pathological environments exists for any safe AGI proposal. 

2. ^{^}

   Ctrl-F"Goal like correlations" here  [LW · GW]

If it is possible to trivially fill in the rest of his argument, then I think it is better for him to post that, instead of posting something that needs to be filled-in, and which doesn't actually back up the thesis that people are interpreting him as arguing for. Precision is a virtue, and I've seen very few essays that actually provide this point about trade explicitly, as opposed to essays that perhaps vaguely allude to the points you have given, as this one apparently does too.

In my opinion, your filled-in argument seems to be a great example of why precision is necessary: to my eye, it contains bald assertions and unjustified inferences about a highly speculative topic, in a way that barely recognizes the degree of uncertainty we have about this domain. As a starting point, why does nanotech imply that it will be cheaper to disassemble humans than to trade with them? Are we assuming that humans cannot fight back against being disassembled, and moreover, is the threat of fighting back being factored into the cost-benefit analysis when the AIs are deciding whether to disassemble humans for their atoms vs. trade with them? Are our atoms really that valuable that it is worth it to pay the costs of violence to obtain them? And why are we assuming that "there will not be other AIs around at the time which 1) would be valuable trade partners for the AI that develops that technology (which gives it that decisive strategic advantage over everyone else) and 2) care about humans at all"?

Satisfying-sounding answers to each of these questions could undoubtedly be given, and I assume you can provide them. I don't expect to find the answers fully persuasive, but regardless of what you think on the object-level, my basic meta-point stands: none of this stuff is obvious, and the essay is extremely weak without the added details that back up its background assumptions. It is very important to try to be truth-seeking and rigorously evaluate arguments on their merits. The fact that this essay is vague, and barely attempts to make a serious argument for one of its central claims, makes it much more difficult to evaluate concretely.

Two reasonable people could read this essay and come away with two very different ideas about what the essay is even trying to argue, given how much unstated inference you're meant to "fill in", instead of plain text that you can read. This is a problem, even if you agree with the underlying thesis the essay is supposed to argue for.

I think this is going to work a bit differently for everyone, and I have no idea what kind of pre-existing math knowledge you have, but part of the answer is: play with it. Let it be fun. Realize that math isn't stuffy rules passed down by wise elders, it's something that came out of people trying to figure out a way to capture the essence of some facet of the world in a way they could work with and manage.

For me, I first really started appreciating math, as opposed to being good at what schools told me math was, when I started reading books like The Art of the Infinite (Written by the Kaplans, who founded The Math Circle, a program to teach kids a creative, problem-solving-based, cooperative approach to math). Other good books to get a flavor of what I mean by "play" might be: Flatterland or anything else for a mass audience by Ian Stewart, anything else for a mass audience by the Kaplans, and if you're up for it, the main popular works by Douglas Hofstadter (Godel Escher Bach, Metamagical Themas, The Mind's I).

Heck, go back and read Alice in Wonderland, but with an eye to the fact that Lewis Caroll was a mathematician writing drug-addled fiction about the advanced math of his day. 

I would even recommend, "Surely You're Joking, Mr. Feynman" for a few really good anecdotes about what it looks like when some actually tries to use their knowledge to operate in the world, as opposed to others who are treating it as some kind of separate magisterium.

And then, try to find little puzzles, things you might never have realized you never knew but that seem like they should be solvable without too much advanced math.

Random examples:

1. Without looking anything up, estimate: how far away is the horizon if you're standing outside on flat ground? On a mountaintop?
2. Imagine you're an ant on a wall in a 10x10x20 room, one foot from the ceiling on the midline of the front wall. You want to go to the midline of the back wall, one foot up from the floor. What's the shortest path, and how long is it?
3. If I have a pile of identical marbles, what's the tightest way to stack/pack them, and how much of the total volume of a space can they fill that way?
4. Do some Fermi problems to get a feel for estimating things. E.g. Go to a library. How many books does it contain? Pages? Word? Letters? What do they all weigh, in total? Or, try to estimate random real-world facts. How many cars or pencils exist?

The reason I think this is important is because "[t]o argue against an idea honestly, you should argue against the best arguments of the strongest advocates": if you write 3000 words inveighing against people who think comparative advantage means that horses can't get sent to glue factories, that doesn't license the conclusion that superintelligence Will Definitely Kill You if there are other reasons why superintelligence Might Not Kill You that don't stop being real just because very few people have the expertise to formulate them carefully.

There's a time for basic arguments, and a time for advanced arguments. I would like to see Eliezer's take on the more complicated arguments you mentioned, but this post is clearly intended to argue basics.

Figuring out whether an RSP is good is hard. You need to consider what high-level capabilities/risks it applies to, and for each of them determine whether the evals successfully measure what they're supposed to and whether high-level risk thresholds trigger appropriate responses and whether high-level risk thresholds are successfully operationalized in terms of low-level evals. Getting one of these things wrong can make your assessment totally wrong. Except that's just for fully-fleshed-out RSPs — in reality the labs haven't operationalized their high-level thresholds and sometimes don't really have responses planned. And different labs' RSPs have different structures/ontologies.

Quantitatively comparing RSPs in a somewhat-legible manner is even harder.

I am not enthusiastic about a recent paper outlining a rubric for evaluating RSPs. Mostly I worry that it crams all of the crucial things—is the response to reaching high capability-levels adequate? are the capability-levels low enough that we'll reach them before it's too late?—into a single criterion, "Credibility." Most of my concern about labs' RSPs comes from those things just being inadequate; again, if your response is too weak or your thresholds are too high or your evals are bad, it just doesn't matter how good the rest of your RSP is. (Also, minor: the "Difficulty" indicator punishes a lab for making ambitious commitments; this seems kinda backwards.)

(I gave up on making an RSP rubric myself because it seemed impossible to do decently well unless it's quite complex and has some way to evaluate hard-to-evaluate things like eval-quality and planned responses.)

(And maybe it's reasonable for labs to not do so much specific-committing-in-advance; I don't know how much better I should feel if OpenAI, DeepMind, and Anthropic made more specific commitments.)

Ok, but you can trivially fill in the rest of it, which is that Eliezer expects ASI to develop technology which makes it cheaper to ignore and/or disassemble humans than to trade with them (nanotech), and that there will not be other AIs around at the time which 1) would be valuable trade partners for the AI that develops that technology (which gives it that decisive strategic advantage over everyone else) and 2) care about humans at all.  I don't think discussion of when and why nation-states go to war with each other is particularly illuminating given the threat model.

I don’t think that at present conditions are good for automating R&D.

I kind of wish this was true, because it would likely mean longer timelines, but my expectation is that the incoming larger LMs + better scaffolds + more inference-time compute could quite easily pass the threshold of significant algorithmic progress speedup from automation (e.g. 2x).