I'll chime in with the same thing I always say: the label "consciousness" is a convenient lumping-together of a bunch of different properties that co-occur in humans, but don't have to all go together a priori.

For example, the way I can do reasoning via subverbal arguments that I can later remember and elaborate if I attend to them. Or what it's like to remember things via associations and some vague indications of how long ago they were, rather than e.g. a discrete system of tags and timestamps. My sense of aversion to noxious stimuli, which can in turn be broken down into many subroutines.

Future AIs are simply going to have some of these properties, to some extent, but not all of them. Asking whether that's consciousness is somewhat useless - what matters is how much of a moral patient we want to treat them as, based on a more fine-grained consideration of their properties.

I agree with the general points and especially "It should be obvious by now that AGI will necessarily be brain-like (in the same ways that DL is brain-like but a bit moreso), and necessarily conscious in human-like ways, as that is simply what intelligence demands".

However, there is much interesting grey area when we start comparing the consciousness of humans with specific types of brain damage to current large transformer AI.

Transformers (even in their more agentic forms) are missing many components of the brain, but their largest deficit is the lack of strong recurrence and medium term memory capacity. Transformer LLMs like GPT3 have an equivalent conscious experience of essentially waking up from scratch, reading around a thousand tokens (in parallel), thinking about those for just a few hundred steps (equivalent to a dozen seconds or so of human thought), and then reset/repeat. They have reasonably extensive short term memory (attention), and very long term memory (standard weights), but not much in between, and they completely lack brain style RNN full recurrence.

In DL terminology brains are massive RNNs with a full spectrum of memory (roughly 10GB equiv in activations, and then a truly massive capacity of 10TB equiv or more in synapses that covers a full spectrum of timescales).

But some humans do have impairments to their medium term memory systems that is perhaps more comparable to LLM transformers - humans with missing/damaged hippocampus/EC regions like the infamous HM. Still conscious, but not in the same way.

Seems like consciousness is rather similar to "the Global Neuronal Workspace", and qualia are rather similar to "what's currently in the Workspace". Is there a reason to reject this way of thinking?

I feel like I already understand, reasonably well, the chain of causation in my brain that leads to me saying the thing in the previous paragraph, i.e. “I’m conscious right now, let me describe the qualia…” See my Book Review: Rethinking Consciousness.

You only have evidence that understand a chain of causation. You don't have evidence that no alternative account is possible.

…And it turns out that there is nothing whatsoever in that chain of causation that looks like what we intuitively expect consciousness and qualia to look like.

If you look at a brain from the outside, its qualia aren't visible. Equally, if you look at your brain from the inside, you see nothing but qualia...you do not see neural activity as such.

And your internal view of causality is that your pains cause you ouches.

Therefore, I need to conclude that either consciousness and qualia don’t exist, or that consciousness and qualia exist, but that they are not the ontologically fundamental parts of reality that they intuitively seem to be.

I don't think qualia seem to be fundamental.

As I understand it, here I’m endorsing the “illusionism” perspective, as advocated (for example) by Keith Frankish, Dan Dennett, and Michael Graziano.

Illusionism is the claim that qualia don't exist at all, not the claim they are merely non-fundamental. An emergentist could agree that they are non-fundamental.

Next, if a computer chip is running similar algorithms as a human philosopher, expressing a similar chain of causation, that leads to that chip emitting similar descriptions of consciousness and qualia as human philosophers emit, for similar underlying reasons, then I think we have to say that whatever consciousness and qualia are (if anything), this computer chip has those things just as much as the human does

1. That isn't illusionism. The most an illusionist would say is that a computer would be subject to the same illusions/delusions.

2. You have bypassed the possibility that what causes qualia to emerge is not computation, but the concrete physics of the brain.... something that can only be captured by a physical description.

(Side note: Transformer-based self-supervised language models like GPT-3 can emit human-sounding descriptions of consciousness, but (I claim) they emit those descriptions for very different underlying reasons than brains do—i.e., as a result of a very different chain of causation / algorithm

Different chain of physical causation , or different algorithm? It's quite possible for the same algorithm to be implemented in physically different ways...and it's quite possible for emergent consciousness to supervene on physics.

However, nihilism is not decision-relevant

Nihilism about what, and why? I don't think you have a theory that consciousness doesn't exist or that qualia don't exist. And even if you did, I don't see how it implies the non existence of values, or preferences or selves or purposes... or whatever else it takes to undermine decision theory.

When I do that, I wind up feeling pretty strongly that if an AGI can describe joy and suffering in a human-like way, thanks to human-like underlying algorithmic processes, then I ought to care about that AGI’s well-being.

Because they have the qualia, or because qualia don't matter?

Because if the agent has to (meta)learn better and better strategies for things like brainstorming and learning and planning and understanding, I think this process entails the kind of self-reflection which comprises full-fledged self-aware human-like consciousness.

Meaning that qualia aren't even one component of human consciousness? Or one possible meaning of "consciousness"?

So I don’t even think the AGI would be in a gray area—I think it would be indisputably conscious, conscious according to any reasonable definition

Illusionist don't think humans are conscious, for some definitions of consciousness.

I feel like I already understand, reasonably well, the chain of causation in my brain that leads to me saying the thing in the previous paragraph

The feeling of understanding, and actual understanding, are very different things. Astrology will give people a feeling of understanding. Popsci books give people a feeling of understanding. Repeating the teacher's password gives a feeling of understanding. Exclaiming "Neurons!" gives people a feeling of understanding. Stories of all sorts give people a feeling of understanding.

One of the signs of real understanding is doing real things with it. If I can build a house that stays up and doesn't leak, I have some understanding of how to build a house. If I can develop a piece of software that performs some practical task, then I have some understanding of software development. If I can help people live better and more fulfilled lives, I have some understanding of people.

Therefore, I need to conclude that either consciousness and qualia don’t exist, or that consciousness and qualia exist, but that they are not the ontologically fundamental parts of reality that they intuitively seem to be.

Or the real explanation is something we have not even thought of yet.

As I understand it, here I’m endorsing the “illusionism” perspective

I don't see how you make the jump from "not ontologically fundamental" to "illusion". For that matter, it's not clear to me what you count as being ontologically fundamental or why it matters.

Honestly Illusionism is just really hard to take seriously. Whatever consciousness is, I have better evidence it exists than anything else since it is the only thing I actually experience directly. I should pretend it isn't real...why exactly? Am I talking to slightly defective P-zombies?

If the computer emitted it for the same reasons...is a clear example of a begging the question fallacy. If a computer claimed to be conscious because it was conscious, then it logically has to be conscious, but that is the possible dispute in the first place. If you claim consciousness isn't real, then obviously computers can't be conscious. Note, that you aren't talking about real illusionism if you don't think we are p-zombies. Only the first of the two possibilities you mentioned is Illusionism if I recall correctly.

You seem like one of the many people trying to systematize things they don't really understand. It's an understandable impulse, but leads to an illusion of understanding (which is the only thing that leads to a systemization like Illusionism seems like frustrated people claiming there is nothing to see here.)
If you want a systemization of consciousness that doesn't claim things it doesn't know, then assume consciousness is the self-reflective and experiential part of the mind that controls and directs large parts of the overall mind. There is no need to state what causes it.

If a machine fails to be self-reflective or experiential then it clearly isn't conscious. It seems pretty clear that modern AI is neither. It probably fails the test of even being a mind in any way, but that's debatable.

Is it possible for a machine to be conscious? Who knows. I'm not going to bet against it, but current techniques seem incredibly unlikely to do it.

Why I expect AGIs to be sentient / conscious, whether we wanted that or not

I think it could be worse than that. 

As you speculate, we will earlier or later figure out how to engineer consciousness. And I mean your strong version of consciousness where people agree that the thing is conscious, e.g., not just because it responds like a conscious agent but because we can point out how it happens, observe the process, and generally see the analogy to how we do it. If we can engineer it, we can optimize it and reduce it to the minimum needed components and computational power to admit consciousness in this sense. Humans got consciousness because it was evolutionary useful or even a side-effect. It is not the core feature of the human brain, and most of the processing and learning of the brain deals with other things. Therefore I conjecture that not that much CPU/RAM will be needed for pure consciousness. I would guess that my laptop has enough for that. What will happen if some evil actor engineers small consciousnesses into their devices or even apps? May we reformat or uninstall them? 

My own point of view is similar to that of Luke Melhausser's opinion in section 4.1 of "2017 report on consciousness and moral patienthood." Physicalism seems like it is true. But the global neural work space + attention schema, and other physicalist models of consciousness, just don't feel like they explain enough of the phenomena, and seem so simple that were they true I'd expect consciousness to be pretty much everywhere. Like, what exactly is reflectivity? What counts as a reflective model for the purpose of generating a computation that outputs thoughts like "I have qualia"? 

1. ^{^}

   Though I suppose where I differ is that I've got philisophical issues with ascribing a computation to a physical process. It seems like you can ascribe many computations to e.g. a rock or a waterfall. And I don't know a philosophical principle which doesn't give you an insane answer and seems justified.  

    

I'm not quite convinced that illusionism is decision-irrelevant in the way you propose. If it's true that there is no such thing as 1st-person experience, then such experience cannot disclose your own values to you. Instead, you must infer your values indirectly through some strictly 3rd-person process. But all external probing of this sort, because it is not 1st-person, will include some non-zero degree of uncertainty.

One paradox that this leads to is the willingness to endure vast amounts of (purportedly illusory) suffering in the hope of winning, in exchange, a very small chance of learning something new about your true values. Nihilism is no help here, because you're not a nihilist; you're an illusionist. You do believe that you have values, instantiated in 3rd-person reality.

You think this will be unpopular now? I gave these fools a decatrain about it in 2014 and they crucified me. Over 10 sentences.

https://www.lesswrong.com/posts/8v5LCvxXcRJELvF4x/ai-is-software-is-ai [LW · GW]

I don't suffer fools. But you, Steven, please feel free to hit me up anytime if you'd like to discuss. This year, next year, whatever. I was born and raised in Boston, just by the way. andywood1 on fb