LessWrong 2.0 Reader

Very well said, I have expressed this exact sentiment more clumsily many times. I concur.

In the absence of AI we can already pass through this phase of realisation/acceptance (the nausea of the realisation of being an object)

Clearly—Sartre was going through it in the early 20th century. I think, while I've never had much trouble with squaring my existence with materialism, I do feel like this period is giving me somewhat of a relapse, and as you say, this is not really within my control.

This feels like the intersection of Analytic and Continental Philosophy

Exactly, I think I was having a continental moment, please do forgive me, we can now return to our scheduled rationality. But really, that's what I'm on about, our claim on rationality has, I guess, been pretty important to me, and I may just be suffering the side effects of seeing being eroded in real time.

Well that is (commendably) the most positive interpretation of the pet hypothesis I've heard. When we think about it, we're really half way there already. By many measures, many of us are living in what anyone in the past would call Utopia, and are very much cared for by many over-arching systems, the market, the government, the internet. I've also never been one to complain "Oh, but what will I do with all my spare time if I don't have to work?"

Perhaps my daughter will have time to reach goals she actually wants to achieve rather than those of necessity. I appreciate your thoughts. 

Why does the plot start at 3x3 instead of 2x2? Of course, it is not common to have games with only one choice, but for Chicken that is what you end up with when removong one option. You could even start the investigation at 2x1 options.

Fair point. From the perspective of one who sees significant value in earning a living and providing goods and services, how are you feeling about the prospects of many marketable skills being mastered by AI? Do we need to reevaluate the value of jobs?

By pointing to a situation where people already don't feel they are contributing much, it seems like Seth is saying that we're not losing much through this rise of AI. But your objection suggests to me you might think that we are losing something significant?

Thanks Seth, I really appreciate what you've said here—it's good to be reminded that it's not necessary to have your kids change the world, and that caring for each other and expressing themselves contributes positively to the whole.

I'm less worried actually about paths my daughter might take, she's very bright and creative and I'm sure she'll do fine, I guess I don't want her to shy away from things just because someone or something else can do it better. I was mainly posting because I felt like that feeling of nausea might be affecting others who keep abreast of these things, and that the coincidence of reading the book right now might have helped me name something others might be experiencing.

But, again, I appreciate your thoughtful insights.

Penrose is quite willing to say that a quantum computer accessing that natural noncomputable dynamics could have the same capabilities as the human brain.

Hmm, we're still talking past each other. I'm saying I don't believe any quantum weirdness is behaviorally relevant to the human brain or simulations thereof. Just ordinary analog and digital computer chips, like in ordinary consumer electronics. Nothing special but the neural architecture and learning rules set up by the genome, and the mundane experience of life.

I kinda think of 'free energy' and 'entropy' as being things that living creatures in some sense 'consume'. We use the 'order' present in the universe to advance our goals (e.g. homeostasis) and leave behind a trail of higher entropy. We harness a gradient of incoming energy and order.

The sunlight which a plant absorbs might counterfactually have been turned into heat after being absorbed by the ground, and ended up in the same entropy state (from the perspective of the universe). Or it might have reflected, traveled light-years through space, and warmed some other thing. The leaf managed to insert itself in this process, intercepting the free energy, and more rapidly-than-counterfactually-expected increased the entropy of the universe.

And living multi-cellular beings are basically made up of tiny entities, cells, which are generally doing the metabolism process internally. And then mitochondria and chloroplasts within cells. But it would be a mistake to say that the living thing is causing itself to be disordered because it's increasing entropy in parts of itself. It's spending free energy (and 'excreting' entropy) in order to accomplish things. For instance, using a muscle (converting some of its stored energy to motion and waste heat) in order to bring food to the creature's mouth. The creature is creating an anti-entropic state, pursuing its specific goals, by increasing the probability of the universe corresponding to its goal state, by causing other things to be extra entropic (always with some extra loss along the way, like from friction). You are missing the order that the agent is creating in the world though if you aren't analyzing the world with the frame of how likely the agent's goals were to be achieved by random chance (e.g. Brownian motion) versus by active optimization efforts by the agent. Anytime a living creature agentically does anything, they are consuming free energy and excreting entropy.

That's my understanding anyway, but I may be using the physics terms wrong since I'm not a physicist.

Not if the point of the argument is to establish that a superintelligence is compatible with achieving the best possible outcome.

Here is a parody of the issue, which is somewhat unfair and leaves out almost all of your argument, but which I hope makes clear the issue I have in mind:

"Proof that a superintelligence can lead to the best possible outcome: Suppose by some method we achieved the best possible outcome. Then, there's no properties we would want a superintelligence to have beyond that, so let's call however we achieved the best possible outcome, 'a superintelligence'. Then, it is possible to have a superintelligence produce the best possible outcome, QED."

In order for an argument to be compelling for the conclusion "It is possible for a superintelligence to lead to good outcomes." you need to use a meaning of "a superintelligence" in the argument, such that the statement "It is possible for a superintelligence to lead to good outcomes", when interpreted with that meaning of "a superintelligence", produces the meaning you want that sentence to have? If I argue "it is possible for a superintelligence, by which I mean computer with a clock speed faster than N, to lead to good outcomes", then, even if I convincingly argue that a computer with a clock speed faster than N can lead to good outcomes, that shouldn't convince people that it is possible for a superintelligence, in the sense that they have in mind (presumably not defined as "a computer with a clock speed faster than N"), is compatible with good outcomes.

Now, in your argument you say that a superintelligence would presumably be some computational process. True enough! If you then showed that some predicate is true of every computational process, you would then be justified in concluding that that predicate is (presumably) true of every possible superintelligence. But instead, you seem to have argued that a predicate is true of some computational process, and then concluded that it is therefore true of some possible superintelligence. This does not follow.

It might be worth knowing that some countries are participating in the "network" without having formal AI safety institutes