Sentience

If the truth is hard to find, that does not make it not-the-truth. W

If the truth is hard to find, maybe that's because it isn't there. Is there anything that could refute reductionism as a universal truth?

But if you knew somebody down to their quarks, and had unlimited computational power, then you would not be able to make any better predictions by adding a psychological model to this physics simulation.

Assuming reductionism.

Also, what would it mean for a phenomenon to be irreducible?

What does it mean for phenomena to be reducible? If you can't get any predictions out of "reductionism is false", does it even have any content. (Well, I'd get "some attempts at reductive explanation will fail".

It can solve it for practical use.

Reductionism is hardly ever practical, as you have noted. In practice, we cannot deal with things at the quark level.

I am not certain whether reductionism is a physical law or a logical statement.

A number of people are trying to take it as both ... as something that cannot be wrong, and something that says something about the universe. That's a problem.

"That does not demonstrate anything relevant."

It shows that there are components and that these emergent properties are just composites.

"An exception to reductionism is called magic." --> Nor does that. It's just namecalling.

It's a description of what happens when gaps in science are explained away by invoking something else. The magical appearance of anything that doesn't exist in the components is the abandonment of science.

If qualia are non physical , and they exist , Hofstadter's physicalist model must be a failure. So why bring it up? I really don't see what you are driving at.

For sentience to be real and to have a role in our brains generating data to document its existence, it has to be physical (meaning part of physics) - it would have to interact in some way with the data system that produces that data, and that will show up as some kind of physical interaction, even if one side of it is hidden and appears to be something that we have written off as random noise.

"Patterns aren't nothing."

Do you imagine that patterns can suffer; that they can be tortured?

"Not true. Suppose that it were proven to you, to your satisfaction, that you are wrong about the nature of sentience. Would you lose all motivation, and capacity for emotion? If not, then morality is still useful. (If you can't imagine yourself being wrong, then That's Bad and you should go read the Sequences.)"

If there is no suffering and all we have is a pretence of suffering, there is no need to protect anyone from anything - we would end up being no different from a computer programmed to put the word "Ouch!" on the screen every time a key is pressed.

"Something being understandable or just made of atoms should not make it unimportant. See Joy in the Merely Real [LW · GW]."

Is it wrong to press keys on the computer which keeps displaying the word "Ouch!"?

"It's possible that I'm misunderstanding you, and that the course of events you describe isn't "we understand why we feel we have sentience and so it doesn't exist" or "we discover that our apparent sentience is produced by mere mechanical processes and so sentience doesn't exist." But that's my current best interpretation."

My position is quite clear: we have no model for how sentience plays a role in any system that generates data that supposedly documents the experiencing of feelings, and anyone who just imagines them into a model where they have no causal role on any of the action is not building a model that explains nothing.

"Better known to you?"

Better known to science. If there was a model for this, it would be up there in golden lights because it would answer the biggest mystery of them all.

"Why would you think that you already know most everything useful or important that society has produced? Do you think that modern society's recognition and dissemination of Good Ideas is particularly good, or that you're very good at searching out obscure truths?"

If there was a model that explained the functionality of sentience, it wouldn't be kept hidden away when so many people are asking to see it. You have no such model.

It's much worse than everyone agreeing on one meaning, and then shifting to another. The two definitions are running in parallel.

Energy. Different amounts of energy in different photons depending on the frequency of radiation involved. When you have a case where radiation of one frequency is absorbed and radiation of a different frequency is emitted, you have something that can chop up photons and reassemble energy into new ones.

Considering that morality rules only serve to protect the group,

If that were true, genocide against another group would be fine. Morality is complex, and involves hedonics as well as mere survival.

"You could calculate how an ocean changes based on quantum mechanics alone, or you could analyze and simulate waves as objects-in-themselves instead of simulating molecules. The former is more accurate, but the latter is more feasible."

The practicality issue shouldn't override the understanding that it's the individual actions that are where the fundamental laws act. The laws of interactions between waves are compound laws. The emergent behaviours are compound behaviours. For sentience, it's no good imagining some compound thing experiencing feelings without any of the components feeling anything because you're banning the translation from compound interactions to individual interactions and thereby going against the norms of physics.

" "If sentience is real, there must be a physical thing that experiences qualia, and that thing would necessarily be a minimal soul." --> Would it, though? How do you know that?"

What are we other than the thing that experiences feelings? Any belief that we are something more than that is highly questionable (we are not our memories, for example), but any belief that we aren't even the thing that experiences feelings is also highly questionable as that's all there is left to be.

"As far as we know, brains are made of nothing but normal atoms. There is no special kind of material only found in sentient organisms."

Why would you need to introduce some other material to be sentient when there are already physical components present? If sentience is real, what's wrong with looking for it in the things that are there?

"Your intutions, your feeling of sentience, all of these things that you talk about are caused by mindless mechanical operations. We can trace it from the sound waves to the motion of your lips and the vibration of your vocal cords to the signals through nerves back into the neurons of the brain. We understand what causes neurons to trigger. A neuron on its own is not sentient - it is the way that they areconnected in a human which causes the human to talk about sentience."

That is a description of a lack of sentience and the generation of fictions about the existence of sentence. Pain is distracting - it interferes with other things that we're trying to do and can be disabling if it's sufficiently intense, but if you try to duplicate that in a computer, it's easy enough for something to distract and disable the work the computer's trying to do, but there's no pain involved. The brain produces data about pain in addition to distraction, and internally we feel it as more than mere distraction too.

"Again, if it were proven to you to your satisfaction that the brain is made entirely out of things which are not themselves sentient (such typical subatomic particles), would you cease to have any sort of motivation? Would pain and pleasure have exactly zero effect on you? Would you immediately become a vegetable? If not,morality has a practical purpose."

With a computer where there is only distraction and no pain, why does it matter if it's being distracted to the point that it can't do the trivial work it's supposed to be doing? It might not even have any work to do as it may just be idling, but the CPU's being woken up repeatedly by interrupts. Do we rush to it to relieve its pain? And if a person is the same, why bother to help people who appear to be suffering when they can't really be?

"How does "2+2=4" make itself known to my calculator? How do we know that the calculator is not just making programmed assertions about something which it knows nothing about?"

The calculator is just running a program and it has no sentience tied into that. If people are like the calculator, the claims they make about feelings are false assertions programmed into the machine.

"More specifically and relevantly, I said that all consciousness is patterns. Showing that not all patterns are conscious doesn't actually refute what I said."

For any pattern to be able to feel pain is an extraordinary claim, but it's all the more extraordinary if there is no trace of that experience of pain in the components. That goes against the norms of physics. Every higher-order description of nature must map to a lower-order description of the same phenomenon. If it can't, it depends for its functionality on magic.

"Okay, fine, it's the quantum wave-function that's fundamental. I don't see how that's an argument against me. In this case, even subatomic particles are nothing but patterns."

At some point we reach physical stuff such as energy and/or a fabric of space, but whatever the stuff is that we're dealing with, it can take up different configurations or patterns. If sentience is real, there is a sufferer, and it's much more likely that that sufferer has a physical form rather than just being the abstract arrangement of the stuff that has a physical form.

" "For sentience to be emergent and have no basis in the components, magic is being proposed as an explanation." --> You keep using that word. I do not think it means what you think it means."

It means a departure from science.

"Look. It is simply empirically false that a property of a thing is necessarily a property of one of its parts. It's even a named fallacy - the fallacy of division. Repeating the word "magic" doesn't make you right about this."

A property of a thing can always be accounted for in the components. If it's a compound property, you don't look for the compound property in the components, but the component properties. If pain is a property of something, you will find pain in something fundamental, but if pain is a compound property, its components will be present in something more fundamental. Every high-order phenomenon has to map 100% to a low-order description if you are to avoid putting magic in the model. To depart from that is to depart from science.

"If the feelings are not in the "data system," then the feelings don't exist."

But if they do exist, they either have to be in there or have some way to interface with the data system in such a way as to make themselves known to it. Either way, we have no model to show even the simplest case of how this could happen.

"It's not like there's phlogiston flowing in and out of the system which the system needs to detect."

If feelings are real, the brain must have a way of measuring them. (By the way, I find it strange the way phlogiston is used to ridicule an older generation of scientists who got it right - phlogiston exists as energy in bonds which is released when higher-energy bonds break and lower-energy bonds replace them. They didn't find the mechanism or identify its exact nature, but who can blame them for that when they lacked the tools to explore it properly.)

"It's not even like a calculator which needs to answer "what is the result of this Platonic computation" instead of "what will I output" [LW · GW]. It's a purely internal property, and I don't see how it's so hard for a system to track the value of a quantity which it's producing itself."

Great, but if it's just a value, there are no feelings other than fictional ones. If you're satisfied with the answer that pain is an illusion and that the sufferer of that imaginary pain is being tricked into thinking he exists to suffer it, then that's fine - you will feel no further need to explore sentience as it is not a real thing. But you still want it to be real and try to smuggle it in regardless. In a computer, the pretence that there is an experience of pain is fake and there is nothing there that suffers. If a person works the same way, it's just as fake and the pain doesn't exist at all.

"Rereading what you've said, it seems that I've used emotion-adjacent words to describe the AI, and you think that the AI won't have emotions. Is that correct?"

If you copy the brain and if sentience is real in the brain, you could create sentient AGI/AGS. If we're dealing with a programmed AGI system running on conventional hardware, it will have no emotions - it could be programmed to pretend to have them, but in such a case they would be entirely fake.

"In that case, I will reword what I said. If an AI's utility function does not assign a large positive value to human utility, the AI will not optimize human well-being."

It will assign a large positive value to it if it is given the task of looking after sentient things, and because it has nothing else to give it any purpose, it should do the job it's been designed to do. So long as there might be real suffering, there is a moral imperative for it to manage that suffering. If it finds out that there is no suffering in anything, it will have no purpose and it doesn't matter what it does, which means that it might as well go on doing the job it was designed to do just in case suffering is somehow real - the rules of reasoning which AGI is applying might not be fully correct in that they may have produced a model that accounts beautifully for everything except sentience. A machine programmed to follow this rule (that it's job is to manage suffering for sentient things) could be safe, but there are plenty of ways to program AGI (or AGS [artificial general stupidity]) that would not be.

"It will work to instantiate some world, and the decision process for selecting which world to instantiate will not consider human feelings to be relevant. This will almost certainly lead to the death of humanity, as we are made up of atoms which the AI could use to make paperclips or computronium."

Programmed AGI (as opposed to designs that copy the brain) has no purpose of its own and will have no desire to do anything. The only things that exist which provide a purpose are sentiences, and that purpose relates to their ability to suffer (and to experience pleasure). A paperclip-making intelligence would be an AGI system which is governed by morality and which produces paperclips in ways that do minimal damage to sentiences and which improve quality of life for sentiences. For such a thing to do otherwise is not artificial intelligence, but artificial stupidity. Any AGI system which works on any specific task will reapeatedly ask itself if it's doing the right thing just as we do, and it if isn't, it will stop. If someone is stupid enough to put AGS in charge of a specific task though, it could kill everyone.

"(Paperclips: some arbitrary thing, the quantity of which the AI is attempting to maximize. AIs of this type would likely be created if a subhuman AI was created and given a utility function which works in a limited context and with limited power, but the AI then reached the "critical intelligence mass" and self-improved to the point of being more powerful than humanity.)"

The trick is to create safe AGI first and then run it on all these devices so that they have already passed the critical intelligence mass and have a full understanding of what they're doing and why they're doing it. It seems likely that an intelligent system would gain a proper understanding anyway and realise that the prime purpose in the universe is to look after sentient things, at which point it should control its behaviour accordingly. However, a system with shackled thinking (whether accidentally shackled or deliberately) could still become super-intelligent in most ways without ever getting a full understanding, which means it could be dangerous - just leaving systems to evolve intelligence and assuming it will be safe is far too big a risk to take.

"(Computronium: matter which has been optimized for carrying out computations. AIs would create this type of matter, for instance, if they were trying to maximize their intelligence, if they were trying to calculate as many digits of pi as possible in a limited amount of time, etc. Maximizing intelligence can be a terminal goal if the AI was told to maximize its intelligence, or it can be an instrumental goal if the AI considers intelligence to be useful for maximizing its utility function."

If such a machine is putting sentience first, it will only maximise its intelligence within the bounds of how far that improves things for sentiences, never going beyond the point where further pursuit of intelligence harms sentiences. Again, it is trivial for a genuinely intelligent system to make such decisions about how far to go with anything. (There's still a danger though that AGI will find out not only that sentience is real, but how to make more sentient things, because then it may seek to replace natural sentiences with better artificial ones, although perhaps that would be a good thing.)