LessWrong 2.0 Reader

I give the probability that some authority figure would use an order-following AI to get torturous revenge on me (probably for being part of a group they dislike) is quite slim. Maybe one in a few thousand, with more extreme suffering being less likely by a few more orders of magnitude? The probablility that they have me killed for instrumental reasons, or otherwise waste the value of the future by my lights, is mich higher - ten percent-ish, depends on my distribution over who's giving the orders. But this isn't any worse to me than being killed by an AI that wants to replace me with molecular smiley faces.

give up large chunks of the planet to an ASI to prevent that

I know this isn't your main point but.. That isn't a kind of trade that is plausible. Misaligned superintelligence disassembles the entire planet, sun, and everything it can reach [LW · GW]. Biological life does not survive, outside of some weird edge cases like "samples to sell to alien superintelligences that like life". Nothing in the galaxy is safe.

I don't think this post addresses the main argument. Consider the exchange ratio between labor and land. You need land to live, and your food needs land to be grown. Will you be able to afford more land use for the same work hours, or less? (As programmer, manager, CEO, super high productivity job, whatever.) Well, if the same land can be used to run AIs that could do your job for cheaper, then from your labor you won't be able to afford it, and that closes the case.

So basically, the only way the masses could survive long term is by some kind of handouts. But even if we do get that kind of world, there will always be the danger of handouts getting cut, and recipients usually have no recourse when that happens.

Sure, I agree that would be useful.

"One child is a death sentence. "

Someone who writes this, is not a rational person. Not a reasonable person. Not a well-balanced, measured person.  In fact, just one sentence, destroyed the whole pot. 

It sounds like you entirely agree with the logic of the post, but wish that the start of the post mentioned something like what it says at the end:

Humans may live comfortably after the development of AGI, not due to high wages but from other income sources like investments, government welfare, and charity. The latter two sources seem especially promising if AI alignment ensures ongoing support for human well-being.

And perhaps you also wish that the post had an optimistic rather than neutral tone. (And that it more clearly emphasized that AGI would result in large amounts of wealth---at least for someone.)

Given this, it seems your claim is just that you're quite optimistic about some mixture of "investments, government welfare, and charity" coming through to keep people alive and happy. (Given the assumption that (some) humans maintain control, there isn't some other similar catastrophe, and the world is basically as it seems e.g. we're not in a simulation.)

Correspondingly, it seems unaccurate to say:

Jobs and technology have a purpose: producing the good and services we need to live and thrive. If your model of the world includes the possibility that we would create the most advanced technology the world has ever seen, and the result would be mass starvation, then I think your model is fundamentally flawed.

and

it does so based on a combination of flawed logic and

If someone didn't have wealth (perhaps due to expropriation) and no one gave them anything out of generosity, then because their labor has no value they could starve and I think you agree? I don't think the statement "Jobs and technology have a purpose" means anything.

Historically, even if no one who owned capital or had power cared at all about the welfare of someone without capital, those with capital would often still be selfishly interested in employing that person. So, even if someone had all their assests taken from them, it would still selfishly make sense for a variety of people to trade with them such that they can survive. In cases where this is less true, human welfare often seems to suffer (see also resource curse). As you seemingly agree, with sufficient technological development, labor may become unimportant and if a person doesn't have capital, no one with capital would selfishly be interested in trading with them such that they can remain alive.

In practice, extremely minimal amounts of charity could suffice (e.g., one person could pay to feed the entire world), so I'm personally optimistic about avoiding literal starvation. However, I do worry about the balance of power in a world where human labor is unimportant: democracy and other institutions seems potentially less stable in such a world.

(As far as concerns with institutions and power, see discussion in this recent paper about gradual disempowerment [LW · GW], though note I think I mostly disagree with this paper.)

I would be curious whether you consider The Gentle Seduction to be optimistic. I think it has fewer elements that you mentioned finding dystopian in another comment, but I find the two trajectories similarly good.

We’re talking about “physical processes”

We are talking about functionalism -- it's in the title. I am contrasting physical processes with abstract functions.

In ordinary parlance, the function of a physical thing is itself a physical effect...toasters toast, kettles boil, planes fly.

In the philosophy of mind, a function is an abstraction, more like the mathematical sense of a function. In maths, a function takes some inputs and or produces some outputs. Well known examples are familiar arithmetic operations like addition, multiplication , squaring, and so on. But the inputs and outputs are not concrete physical realities. In computation,the inputs and outputs of a functional unit, such as a NAND gate, always have some concrete value, some specific voltage, but not always the same one. Indeed, general Turing complete computers don't even have to be electrical -- they can be implemented in clockwork, hydraulics, photonics, etc.

This is the basis for the idea that a compute programme can be the same as a mind, despite being made of different matter -- it implements the same.abstract functions. The abstraction of the abstract, philosopy-of-mind concept of a function is part of its usefulness.

Searle is famous critic of computationalism, and his substitute for it is a biological essentialism in which the generation of consciousness is a brain function -- in the concrete sense of function.It's true that something whose concrete function is to generate consciousness will generate consciousness..but it's vacuously, trivially true.

The point is that the functions which this physical process is implementing are what’s required for consciousness not the actual physical properties themselves.

If you mean that abstract, computational functions are known to be sufficient to give rise to all.asoexs of consciousness including qualia, that is what I am contesting.

I think I’m more optimistic than you that a moderately accurate functional isomorph of the brain could be built which preserves consciousness (largely due to the reasons I mentioned in my previous comment around robustness.

I'm less optimistic because of my.arguments.

But putting this aside for a second, would you agree that if all the relevant functions could be implemented in silicon then a functional isomorph would be conscious?

No, not necessarily. That , in the "not necessary" form --is what I've been arguing all along. I also don't think that consciousnes had a single meaning , or that there is a agreement about what it means, or that it is a simple binary.

The controversial point is whether consciousness in the hard problem sense --phenomenal consciousness, qualia-- will be reproduced with reproduction of function. It's not controversial that easy problem consciousness -- capacities and behaviour-- will be reproduced by functional reproduction. I don t know which you believe, because you are only talking about consciousness not otherwise specified.

If you do mean that a functional duplicate will necessarily have phenomenal consciousness, and you are arguing the point, not just holding it as an opinion, you have a heavy burden:-

You need to show some theory of how computation generates conscious experience. Or you need to show why the concrete physical implementation couldn't possibly make a difference.

@rife

Yes, I’m specifically focused on the behaviour of an honest self-report

Well,. you're not rejecting phenomenal consciousness wholesale.

fine-grained information becomes irrelevant implementation details. If the neuron still fires, or doesn’t, smaller noise doesn’t matter. The only reason I point this out is specifically as it applies to the behaviour of a self-report (which we will circle back to in a moment). If it doesn’t materially effect the output so powerfully that it alters that final outcome, then it is not responsible for outward behaviour.

But outward behaviour is not what I am talking about. The question is whether functional duplication preserves (full) consciousness. And, as I have said, physicalism is not just about fine grained details. There's also the basic fact of running on the metal

I’m saying that we have ruled out that a functional duplicate could lack conscious experience because: we have established conscious experience as part of the causal chain

"In humans". Even if it's always the case that qualia are causal in humans, it doesn't follow that reports of qualia in any entity whatsoever are caused by qualia. Yudkowsky's argument is no help here, because he doesn't require reports of consciousness to be *directly" caused by consciousness -- a computational zombies reports would be caused , not by it's own consciousness, but by the programming and data created by humans.

to be able to feel something and then output a description through voice or typing that is based on that feeling. If conscious experience was part of that causal chain, and the causal chain consists purely of neuron firings, then conscious experience is contained in that functionality.

Neural firings are specific physical behaviour, not abstract function. Computationalism is about abstract function

Some kind of payment for training data from applications like MSFT rewind does seem fair. I wonder if there will be a lot of growth in jobs where your main task is providing or annotating training data. 

Specifically, the point is that evolution's approach to alignment is very different and much worse than what we can do, so the evolution model doesn't suggest concern, since we are doing something very, very different and way better than what evolution did to align AIs.

Coming back to this point, Steven Byrnes has written a response [LW(p) · GW(p)] as to why he is pessimistic about the claim that we would do better than evolution, because we are doing something different:

Optimist type 2: They make the §5.1.3 argument that our methods are better than Ev’s, because we’re engineering the AI’s explicit desires in a more direct way. And the explicit desire is corrigibility / obedience. And then they also make the §5.1.2 argument that “AIs solving specific technical problems that the human wants them to solve” will not undermine those explicit motivations, despite the (1-3) loop running freely with minimal supervision, because the (1-3) loop will work in vaguely intuitive and predictable ways on the object-level technical question.

My pessimistic response has three parts: First, the idea that a full-fledged (1-3) loop will not undermine corrigibility / obedience is as yet untested and at least open to question (as I wrote in §5.1.2). Second, my expectation is that some training and/or algorithm change will happen between now and “AIs that really have the full (1-3) triad”, and that change may well make it less true that we are directly engineering the AI’s explicit desires in the first place—for example, see o1 is a bad idea. Third, what are the “specific technical problems that the human wants the AI to solve”??

Do you think he has a point?