Is it desirable for the first AGI to be conscious?
post by Charbel-Raphaël (charbel-raphael-segerie) · 2022-05-01T21:29:55.103Z · LW · GW · 7 commentsThis is a question post.
Contents
Answers 3 grantcastillou@gmail.com 1 Raphaël S None 7 comments
It seems that consciousness is one of the essential things necessary to characterize the moral status of an agent.
It seems that we have very little chance to solve the AI safety problem.
We don't know if the first AGI will be conscious.
Since we are in a certain sense doomed [LW · GW], the future consists of the following 2 cases:
- Either the first AGI is not sentient and never becomes sentient.
- Or the first AGI acquires sentience at a given moment.
- Either the valence of this AGI is positive
- Or the valence of this AGI is negative
- Either the valence is mixed
- Or the AGI reprograms itself to experience a positive valence (instrumental convergence if its well-being is included in the utility function?)
I have no idea which scenario is the most desirable. The uncertainty is multiplied by considering the AGI's superior capabilities, and its ability to multiply. So perhaps we could see it as a utility monster. Therefore, the following questions seem very important:
(1) What do you think is the most likely scenario and (2) the most desirable scenario?
(3|2) Conditional on the scenario that seems most desirable to you, is there a way to steer the future in that direction?
Answers
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with primary consciousness will probably have to come first. The thing I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing. I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order. My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461
↑ comment by Charbel-Raphaël (charbel-raphael-segerie) · 2022-05-05T00:09:44.465Z · LW(p) · GW(p)
"and higher order consciousness, which came to only humans with the acquisition of language. A machine with primary consciousness will probably have to come first"
The language models like GPT3 appear to be unstoppable, and are much more versatile than the models of other modalities.
So maybe a machine with higher order consciousness will come first?
↑ comment by Charbel-Raphaël (charbel-raphael-segerie) · 2022-05-05T00:12:24.867Z · LW(p) · GW(p)
The difference between primary consciousness and higher order consciousness is not clear to me. Is it the same thing as access consciousness and meta cognition?
↑ comment by Charbel-Raphaël (charbel-raphael-segerie) · 2022-05-05T11:20:13.899Z · LW(p) · GW(p)
I just discovered this debate thanks to a YouTube recommendation from the Institute for the Future of Life. I find the formulation of the question by Anderw Serazin very well put.
The Yes Team: We would identify with it better, and it would help with creativity. You would want your digital assistant to be able to help you by modeling you, it seems necessary to model the human.
No team: It's easier to deal with IA safety without it, the possibility to create infinite suffering is bad, unpleasant task in the world are better done by non-conscious agent.
I think the no team is right.
Interesting: during the questions, Yoshua Bengio advocates decoupling moral status from subjective experience (Moral status: would be the role in society of the entity !?). And then he proposes the following taxonomy of the concept of consciousness:
- Subjective experience: trivially already incorporated in modern deep learning because each neural network has learned its own representation of the world.
- self-awareness: useful notion for the agent who moves in the world
- emotion: already present in reinforcement learning in a primitive way.
7 comments
Comments sorted by top scores.
comment by Shmi (shminux) · 2022-05-01T23:39:26.607Z · LW(p) · GW(p)
How can one tell if an AI is conscious/sentient/sapient?
Replies from: charbel-raphael-segerie↑ comment by Charbel-Raphaël (charbel-raphael-segerie) · 2022-05-02T11:24:02.706Z · LW(p) · GW(p)
I agree that today the concept of consciousness is very poorly defined, but I think that in the future it will be possible to define it in a way that will make sense, or at least that we will be able to correct our current intuitions.
How can one tell if a human is conscious?
In humans, we have clues [LW · GW]. For example, it is possible to experiment by varying the duration of a light stimulus. There are stimuli of very short duration that are processed by visual areas V1 and V2, but which do not go up to the parietal cortex. For stimuli of slightly longer duration, the light spot induces a conscious response. The subject can then say: yes, I saw the spot. In humans, the study can be done on a declarative basis.
How can one tell if an AI is conscious?
At present, such an experiment cannot be performed on GPT (unless an analogy is made between the duration of a stimulus for humans and the activation in the attention layers of the transformer?)
Indications of consciousness in an AI would be:
- A global workspace, which allows a central piece of information to be accessible from anywhere in the system (e.g. in the schema of the paper attention is all you need, the link between the last encoder and all decoders, or recently in the Socratic models, an implementation of a common language module)
- consciousness seems to be a way of processing information in a complete Turing fashion, but it is quite slow, whereas unconscious processes automatically a multitude of bits in parallel. So I would guess another clue would be a distinction between several modes of functioning [? · GW]:
- an automatic mode, system 1, such as GPU parallelized vision networks today
- vs a slower mode, system 2, which would be a very general program which cannot be parallelized and should be run on cpu.
- Perhaps an implementation of metacognition or a reflexive system: parts of the neural network that attempt to predict the future or current state of other parts of the neural network. But this point depends on whether one considers the mirror test to be necessary for consciousness.
- ?
Replies from: TAG↑ comment by TAG · 2022-05-04T17:57:25.712Z · LW(p) · GW(p)
How can one tell if a human is conscious?
That's the easy case ..each of us we can tell we are conscious by introspection, and a normal-seeming and behaving person isn't that different.
Replies from: charbel-raphael-segerie↑ comment by Charbel-Raphaël (charbel-raphael-segerie) · 2022-05-05T00:04:59.711Z · LW(p) · GW(p)
Yeah, I know, I just wanted to begin answering with this and to present in one sentence (and without mentioning it, my bad...) the concept of neural correlates of consciousness
comment by Dagon · 2022-05-02T16:26:27.805Z · LW(p) · GW(p)
Are you using "conscious" and "sentient" as synonyms? Do you think some, most, or all humans are conscious? Why do you think this?
I'm not sure what it means for it to be "desirable" for an entity to be conscious. From a utilitarian standpoint, there's some controversy on how to account for size of population even in humans or animals, but it's generally accepted that if something IS conscious, then it's better for that thing to have more positive experiences.
Replies from: TAG, charbel-raphael-segerie↑ comment by Charbel-Raphaël (charbel-raphael-segerie) · 2022-05-02T16:57:52.120Z · LW(p) · GW(p)
Are you using "conscious" and "sentient" as synonyms? → Yes. Maybe sentient is consciousness with only valence but it is basically the same thing. But you're right, I shouldn't have used both worlds at the same time without further explanation.
Do you think some, most, or all humans are conscious? → Yes, I think most humans are conscious. Why do you think this? Because I am conscious, but I'm not sure if I really answer your question here.
"I'm not sure what it means for it to be "desirable" for an entity to be conscious." → Yes this is a good question!
- Is it desirable for a stone to be conscious? Um, yeah. I'm not sure what that means.
- Is it desirable for a human to be visually conscious? Yes. I don't want you to become Blindsight.
- Is it desirable for a human to be conscious? By extension, yes, I think so.