Posts
Comments
I think you are very confused about how to interpret disagreements around which mental processes ground consciousness. These disagreements do not entail a fundamental disagreement about what consciousness is as a phenomenon to be explained.
Regardless of that though, I just want to focus on one of your "referents of consciousness" here, because I also think the reasoning you provide for your particular claims is extremely weak. You write the following
#9: Symbol grounding. Even within a single interaction, an LLM can learn to associate a new symbol to a particular meaning, report on what the symbol means, and report that it knows what the symbol means.
The behavioural capacity you describe does not suffice for symbol grounding. Indeed, the phrase "associate a new symbol to a particular meaning" begs the question, because the symbol grounding problem asks how it is that symbolic representations in computing systems can acquire meaning in the first place.
The most famous proposal for what it would take to ground symbols was proposed by Harnad in his classic 1990 paper. Harnad thought grounding was sensorimotor in nature and required both iconic and categorical representations, where iconic representations are formed via "internal analog transforms of the projections of distal objects on our sensory surfaces" and categorical representations are "those 'invariant features' of the sensory projection that will reliably distinguish a member of a category from any nonmembers" (Harnad though connectionist networks were good candidates for forming such representations and time has shown him to be right). Now, it seems to me highly unlikely that LLMs exhibit their behavioural capacities in virtue of iconic representations of the relevant sort, since they do not have "sensory surfaces" in anything like the right kind of way. Perhaps you disagree, but merely describing the behavioural capacities is not evidence.
Notably, Harnad's proposal is actually one of the more minimal answers in the literature to the symbol grounding problem. Indeed, he received significant criticism for putting the bar so low. More demanding theorists have posited the need for multimodal integration (including cognitive and sensory modalities), embodiment (including external and internal bodily processes), normative functions, a social environment and more besides. See Barsalou for a nice recent discussion and Mollo and Milliere for an interesting take on LLMs in particular.
I don't think so. Compare the following two requests:
(1) Describe a refrigerator without using the word refrigerator or near-synonyms.
(2) Describe the structure of a refrigerator in terms of moving parts and/or subprocesses.
The first request demands the tabooing of words; the second request demands an answer of a particular (theory-laden) form. I think the OPs request is like request 2. What's more, I expect submitting request 2 to a random sample of people would license the same erroneous conclusion about "refrigerator" as it did about "consciousness".
This is not to say there are no special challenges associated with "consciousness" that do not hold for "refrigerator". Indeed, I believe there are. However, the basic point that people can be referring to a single phenomenon even if they have different beliefs about the phenomenon's underlying structure seems to me fairly straightforward.
Edit: I see sunwillrise gave a much more detailed response already. That response seems pretty much on the money to me.
I will also point people to this paper if they are interested in reading an attempt by a prominent philosopher of consciousness at defining it in minimally objectionable terms.
Section 1.6 is another appendix about how this series relates to Philosophy Of Mind. My opinion of Philosophy Of Mind is: I’m against it! Or rather, I’ll say plenty in this series that would be highly relevant to understanding the true nature of consciousness, free will, and so on, but the series itself is firmly restricted in scope to questions that can be resolved within the physical universe (including physics, neuroscience, algorithms, and so on). I’ll leave the philosophy to the philosophers.
At the risk of outing myself as a thin-skinned philosopher, I want to push back on this a bit. If we are taking "philosophy of mind" to mean, "the kind of work philosophers of mind do" (which I think we should), then your comment seems misplaced. Crucially, one need not be defending particular views on "big questions" about the true nature of consciousness, free will, and so on to be doing philosophy of mind. Rather, much of the work philosophers of mind do is continuous with scientific inquiry. Indeed, I would say some philosophy of mind is close to indistinguishable from what you do in this post! For example, lots of this work involves trying to carve up conceptual space in a way that coheres with empirical findings, suggests avenues for further research, and renders fruitful discussion easier. Your section 1.3 in this post features exactly the kind of conceptual work that is the bread-and-butter of philosophy. So, far from leaving philosophy to the philosophers, I actually think your work would fit comfortably into the more empirically informed end of contemporary philosophy of mind. To end on a positive note, I think it's really clearly written, fascinating, and fun to read. So thanks!
I strongly believe that step 1 is sufficient or almost sufficient for step 2, i.e., that it's impossible to give an adequate account of human phenomenology without figuring out most of the computational aspects of consciousness.
Apologies for nitpicking, but your strong belief that step 1 is (almost) sufficient for step 2 would be more faithfully re-phrased as: it will (probably) be possible/easy to give an adequate account of human phenomenology by figuring out most of the computational aspects of consciousness. The way you phrased it (viz., "impossible...without") is equivalent to saying that step 1 is necessary for step 2, an importantly different claim (on this phrasing, something besides the computational aspects may be required). Of course, you may think it is both necessary and sufficient, I'm just pointing out the distinction.
I agree with the thrust of this comment, which I read as saying something like "our current physics is not sufficient to explain, predict, and control all macroscopic phenomena". However, this is a point which Sean Carroll would agree with. From the paper under discussion (p.2): "This is not to claim that physics is nearly finished and that we are close to obtaining a Theory of Everything, but just that one particular level in one limited regime is now understood."
The claim he is making, then, is totally consistent with the need to find further approximations and abstractions to model macroscopic phenomena. His point is that none of that will dictate modifications to the core theory (effective quantum field theory) when applied to "everyday" phenomena which occur in regions of the universe which we currently interact with (because the boundary conditions of this region of the universe are compatible with EQFT). Another way to put this is that Carroll claims no possible experiment can be conducted within the "everyday regime" which will falsify the core theory. Do you still disagree?
For the record, this is just to clarify what Carroll's claim is. I totally agree that that none of this is relevant to overcoming the limitations of formal verification, which very clearly depend on many abstractions and approximations and will continue to do so for the foreseeable future.
I see. I'm afraid I don't have much great literature to recommend on computational semantics (though Josh Tenenbaum's PhD dissertation seems relevant). I still wonder whether, even if you disagree with the approaches you have seen in that domain, those might be the kind of people well-placed to help with your project. But that's your call of course.
Depending on your goals with this project, you might get something out of reading work by relevance theorists like Sperber, Wilson, and Carston (if you haven't before). I find Carston's reasoning about how various aspects of language works quite compelling. You won't find much to help solve your mathematical problems there, but you might find considerations that help you disambiguate between possible things you want your model of semantics to do (e.g., do you really care about semantics, per se, or rather concept formation?).
Thanks for the response. Personally, I think your opening sentence as written is much, much too broad to do the job you want it to do. For example, I would consider "natural language semantics as studied in linguistics" to include computational approaches, including some Bayesian approaches which are similar to your own. If I were a computational linguist reading your opening sentence, I would be pretty put off (presumably, these are the kind of people you are hoping not to put off). Perhaps including a qualification that it is classical semantics you are talking about (with optional explanatory footnote) would be a happy medium.
I enjoyed the content of this post, it was nicely written, informative, and interesting. I also realise that the "less bullshit" framing is just a bit of fun that shouldn't be taken too seriously. Those caveats aside, I really dislike your framing and want to explain why! Reasons below.
First, the volume of work on "semantics" in linguistics is enormous and very diverse. The suggestion that all of it is bullshit comes across as juvenile, especially without providing further indication as to what kind of work you are talking about (the absence of a signal that you are familiar with the work you think is bullshit is a bit galling).
Second, this work might interest people who work on similar things. Indeed, this seems like something you are explicitly after. However, your casual dismissal of prior work on semantics as bullshit combined with a failure to specify the nature of the project you are pursuing in terms a linguist would recognise (i.e., your project is far more specific than "semantics") could prevent engagement and useful feedback from the very people who are best-placed to provide it.
Third, on the object level, I think there is a gulf in numerosity (from more to less numerous) separating 1) human concepts (these might roughly be the "latent variables in probabilistic generative models" in your and Steven Byrnes comment chain), 2) communicable human concepts (where communicability is some kind of equivalence, as in your model), and 3) human concepts with stable word meanings in the current lexicon (like common nouns). I think your framing in this post conflates the three (even if you yourself do not). The reason I include this object level worry here is that, if my worry is indicative of how others might react, then it could be more of a turn off for potential collaborators to see these notions conflated in the same post as deriding other work as not capturing what semantics is really about (if you think my distinctions are reasonable, which of them do you think semantics is really about and what exactly do you worry linguists have been erroneously working on all this time?).
Again, interesting work, hope this didn't come off too combative!
Fair enough if literally any approach using symbolic programs (e.g. a python interpreter) is considered neurosymbolic, but then there isn't any interesting weight behind the claim "neurosymbolic methods are necessary".
If somebody achieved a high-score on the ARC challenge by providing the problems to an LLM as prompts and having it return the solutions as output, then the claim "neurosymbolic methods are necessary" would be falsified. So there is weight to the claim. Whether it is interesting or not is obviously in the eye of the beholder.
I think the kind of sensible goalpost-moving you are describing should be understood as run-of-the-mill conceptual fragmentation, which is ubiquitous in science. As scientific communities learn more about the structure of complex domains (often in parallel across disciplinary boundaries), numerous distinct (but related) concepts become associated with particular conceptual labels (this is just a special case of how polysemy works generally). This has already happened with scientific concepts like gene, species, memory, health, attention and many more.
In this case, it is clear to me that there are important senses of the term "general" which modern AI satisfies the criteria for. You made that point persuasively in this post. However, it is also clear that there are important senses of the term "general" which modern AI does not satisfy the criteria for. Steven Byrnes made that point persuasively in his response. So far as I can tell you will agree with this.
If we all agree with the above, the most important thing is to disambiguate the sense of the term being invoked when applying it in reasoning about AI. Then, we can figure out whether the source of our disagreements is about semantics (which label we prefer for a shared concept) or substance (which concept is actually appropriate for supporting the inferences we are making).
What are good discourse norms for disambiguation? An intuitively appealing option is to coin new terms for variants of umbrella concepts. This may work in academic settings, but the familiar terms are always going to have a kind of magnetic pull in informal discourse. As such, I think communities like this one should rather strive to define terms wherever possible and approach discussions with a pluralistic stance.
I actually think what you are going for is closer to JL Austin's notion of an illocutionary act than anything in Wittgenstein, though as you say, it is an analysis of a particular token of the type ("believing in"), not an analysis of the type. Quoting Wikipedia:
"According to Austin's original exposition in How to Do Things With Words, an illocutionary act is an act:
- (1) for the performance of which I must make it clear to some other person that the act is performed (Austin speaks of the 'securing of uptake'), and
- (2) the performance of which involves the production of what Austin calls 'conventional consequences' as, e.g., rights, commitments, or obligations (Austin 1975, 116f., 121, 139)."
Your model of "believing in" is essentially an unpacking of the "conventional consequences" produced by using the locution in various contexts. I think it is a good unpacking, too!
I do think that some of the contrasts you draw (belief vs. believing in) would work equally well (and with more generality) as contrasts between beliefs and illocutionary acts, though.
In Leibniz’ case, he’s known almost exclusively for the invention of calculus.
Was this supposed to be a joke (if so, consider me well and truly whooshed)? At any rate, it is most certainly not the case. Leibniz is known for a great many things (both within and without mathematics) as can be seen from a cursory glance at his Wikipedia page.
Rather, they might be mere empty machines. Should you still tolerate/respect/etc them, then?"
My sense is that I'm unusually open to "yes," here.
I think the discussion following from here is a little ambiguous (perhaps purposefully so?). In particular, it is unclear which of the following points are being made:
1: Sufficient uncertainty with respect to the sentience (I'm taking this as synonymous with phenomenal consciousness) of future AIs should dictate that we show them tolerance/respect etc...
2: We should not be confident that sentience is a good criterion for moral patienthood (i.e., being shown tolerance/respect etc...), even though sentience is a genuine thing.
3: We should worry that sentience isn't a genuine thing at all (i.e, illusionism / as-yet-undescribed re-factorings of what we currently call sentience).
When you wrote that you are unusually open to "yes" in the quoted sentence, I took the qualifier "unusual" to indicate that you were making point 2, since I do not consider point 1 to be particularly unusual (Schwitzgebel has pushed for this view, for example). However, your discussion then mostly seemed to be making the case for point 1 (i.e., we could impose a criterion for moral worth that is intended to demarcate non-sentient and sentient entities but that fails). For what it's worth, I would be very interested to hear arguments for point 2 which do not collapse into point 1 (or, alternatively, some reason why I am mistaken for considering them distinct points). From my perspective, it is hard to understand how something which really lacks what I mean by phenomenal consciousness could possibly be a moral patient. Perhaps it is related to the fact that I have, despite significant effort, utterly failed to grok illusionism.
Apologies, I had thought you would be familiar with the notion of functionalism. Meaning no offence at all but it's philosophy of mind 101, so if you're interested in consciousness, it might be worth reading about it. To clarify further, you seem to be a particular kind of computational functionalist. Although it might seem unlikely to you, since I am one of those "masturbatory" philosophical types who thinks it matters how behaviours are implemented, I am also a computational functionalist! What does this mean? It means that computational functionalism is a broad tent, encompassing many different views. Let's dig into the details of where we differ...
If something can talk, then, to a functionalist like me, that means it has assembled and coordinated all necessary hardware and regulatory elements and powers (that is, it has assembled all necessary "functionality" (by whatever process is occurring in it which I don't actually need to keep track of (just as I don't need to understand and track exactly how the brain implements language))) to do what it does in the way that it does.
This is a tautology. Obviously anything that can do a thing ("talk") has assembled the necessary elements to do that very thing in the way that it does. The question is whether or not we can make a different kind of inference, from the ability to implement a particular kind of behaviour (linguistic competence) to the possession of a particular property (consciousness).
Once you are to the point of "seeing something talk fluently" and "saying that it can't really talk the way we can talk, with the same functional meanings and functional implications for what capacities might be latent in the system" you are off agreeing with someone as silly as Searle. You're engaged in some kind of masturbatory philosophy troll where things don't work and mean basically what they seem to work and mean using simple interactive tests.
Okay, this is the key passage. I'm afraid your view of the available positions is seriously simplistic. It is not the case that anybody who denies the inference from 'displays competent linguistic behaviour' to 'possesses the same latent capacities' must be in agreement with Searle. There is a world of nuance between your position and Searle's, and most people who consider these questions seriously occupy the intermediate ground.
To be clear, Searle is not a computational functionalist. He does not believe that non-biological computational systems can be conscious (well, actually he wrote about "understanding" and "intentionality", but his arguments seem to apply to consciousness as much or even more than they do to those notions). On the other hand, the majority of computational functionalists (who are, in some sense, your tribe) do believe that a non-biological computational system could be conscious.
The variation within this group is typically with respect to which computational processes in particular are necessary. For example, I believe that a computational implementation of a complex biological organism with a sufficiently high degree of resolution could be conscious. However, LLM-based chatbots are nowhere near that degree of resolution. They are large statistical models that predict conditional probabilities and then sample from them. What they can do is amazing. But they have little in common with living systems and only by totally ignoring everything except for the behavioural level can it even seem like they are conscious.
By the way, I wouldn't personally endorse the claim that LLM-based chatbots "can't really talk the way we talk". I am perfectly happy to adopt a purely behavioural perspective on what it means to "be able to talk". Rather, I would deny the inference from that ability to the possession of consciousness. Why would I deny that? For the reasons I've already given. LLMs lack almost all of the relevant features that philosophers, neuroscientists, and biologists have proposed as most likely to be necessary for consciousness.
Unsurprisingly, no, you haven't changed my mind. Your claims require many strong and counterintuitive theoretical commitments for which we have either little or no evidence. I do think you should take seriously the idea that this may explain why you have found yourself in a minority adopting this position. I appreciate that you're coming from a place of compassion though, that's always to be applauded!
I am sorry that you got the impression I was trolling. Actually I was trying to communicate to you. None of the candidate criteria I suggested were conjured ex nihilo out of a hat or based on anything that I just made up. Unfortunately, collecting references for all of them would be pretty time consuming. However, I can say that the global projection phrase was gesturing towards global neuronal workspace theory (and related theories). Although you got the opposite impression, I am very familiar with consciousness research (including all of the references you mentioned, though I will admit I don't think much of IIT).
The idea of "meat chauvinism" seems to me a reductive way to cast aside the possibility that biological processes could be relevant to consciousness. I think this is a theoretical error. It is not the case that taking biological processes seriously when thinking about consciousness implies (my interpretation of what you must mean by) "meat chauvinism". One can adopt a functionalist perspective on biological processes that operate far below the level of a language production system. For example, one could elaborate a functional model of metabolism which could be satisfied by silicon-based systems. In that sense, it isn't meat chauvinism to suggest that various biological processes may be relevant to consciousness.
This would discount human uploads for silly reasons. Like if I uploaded and was denied rights for lack of any of these things they I would be FUCKING PISSED OFF
Assuming what you mean by "fucking pissed off" involves subjective experience and what you mean by "I was uploaded" would not involve implementing any of the numerous candidates for necessary conditions on consciousness that I mentioned, this is simply begging the question.
To me, it doesn't make any sense to say you would have been "uploaded" if all you mean is a reasonably high fidelity reproduction of your input-output linguistic behaviour had been produced. If what you mean by uploaded is something very different which would require numerous fundamental scientific breakthroughs then I don't know what I would say, since I don't know how such an upload would fare with respect to the criteria for conscious experience I find most compelling.
Generally speaking, there is an enormous difference between hypothetical future simulated systems of arbitrary sophistication and the current generation of LLM-based chatbots. My sense is that you are conflating these things when assessing my arguments. The argument is decidedly not that the LLM-based chatbots are non-biological, therefore they cannot be conscious. Nor is it that no future silicon-based systems, regardless of functional organisation, could ever be conscious. Rather, the argument is that LLM-based chatbots lack almost all of the functional machinery that seems most likely to be relevant for conscious experience (apologies that using the biological terms for these aspects of functional machinery was misleading to you), therefore they are very unlikely to be conscious.
I agree that the production of coherent linguistic output in a system that lacks this functional machinery is a scientific and technological marvel, but it is only evidence for conscious experience if your theory of consciousness is of a very particular and unusual variety (relative to the fields which study the topic in a professional capacity, perhaps such ideas have greater cache on this website in particular). Without endorsing such a theory, the evidence you provide from what LLMs produce, given their training, does not move me at all (we have an alternative explanation for why LLMs produce such outputs which does not route through them being subjectively experiencing entities, and what's more, we know the alternative explanation is true, because we built them).
Given how you responded above, I have the impression you think neuroscience and biology are not that relevant for understanding consciousness. Clearly, I disagree. May I ask what has given you the impression that the biological details don't matter (even when given a functional gloss such that they may be implemented in silico)?
I think you're missing something important.
Obviously I can't speak to the reason there is a general consensus that LLM-based chatbots aren't conscious (and therefore don't deserve rights). However, I can speak to some of the arguments that are still sufficient to convince me that LLM-based chatbots aren't conscious.
Generally speaking, there are numerous arguments which essentially have the same shape to them. They consist of picking out some property that seems like it might be a necessary condition for consciousness, and then claiming that LLM-based chatbots don't have that property. Rather than spend time on any one of these arguments, I will simply list some candidates for such a property (these may be mentioned alone or in some combination):
Metabolism, Temporally continuous existence, Sensory perception, Integration of sensory signals, Homeostatic drives, Interoception, Coherent self-identity, Dynamic coupling to the environment, Affective processes, A nervous system, Physical embodiment, Autonomy, Autopoiesis, Global projection of signals, Self-monitoring, Synchronized neuronal oscillations, Allostasis, Executive function, Nociceptors, Hormones... I could keep going.
Naturally, it may that some of these properties are irrelevant or unnecessary for consciousness. Or it could be that even altogether they are insufficient. However, the fact that LLM-based chatbots possess none of these properties is at least some reason to seriously doubt that they could be conscious.
A different kind of argument focuses more directly on the grounds for the inference that LLM-based chatbots might be conscious. Consider the reason that coherent linguistic output seems like evidence of consciousness in the first place. Ordinarily, coherent linguistic output is produced by other people and suggests consciousness to us based on a kind of similarity-based reasoning. When we encounter other people, they are engaging in similar behaviour to us, which suggests they might be similar to us in other ways, such as having subjective experience. However, this inference would no longer be justified if there was a known, significantly different reason for a non-human entity to produce coherent linguistic output. In the case of LLM-based chatbots, we do have such a reason. In particular, the reason is the data-intensive training procedure, a very different story for how other humans come to learn to produce coherent linguistic output.
Nobody should be 100% confident in any claims about which entities are or are not conscious, but the collective agreement that LLM-based chatbots are not seems pretty reasonable.
Enjoyable post, I'll be reading the rest of them. I especially appreciate the effort that went into warding off the numerous misinterpretations that one could easily have had (but I'm going to go ahead an ask something that may signal I have misinterpreted you anyhow).
Perhaps this question reflects poor reading comprehension, but I'm wondering whether you are thinking of valence as being implemented by something specific at a neurobiological level or not? To try and make the question clearer (in my own head as much as anything), let me lay out two alternatives to having valence implemented by something specific. First, one might imagine that valence is an abstraction over the kind of competitive dynamics that play out among thoughts. On this view, valence is a little like evolutionary fitness (the tautology talk in 1.5.3 brought this comparison to mind). Second, one might imagine that valence is widely distributed across numerous brain systems. On this view, valence is something like an emotion (if you'll grant the hopefully-no-longer-controversial claim that the neural bases of emotions are widely distributed). I don't think either of these alternatives are what you are going for, but I also didn't see the outright claim that valence is something implemented by a specific neurobiological substrate. What do you believe?
In other words, you think that even in a world where the distribution of mathematical methods were very specific to subject areas, this methodology would have failed to show that? If so, I think I disagree (though I agree the evidence of the paper is suggestive, not conclusive). Can you explain in more detail why you think that? Just to be clear, I think the methodology of the paper is coarse, but not so coarse as to be unable to pick out general trends.
Perhaps to give you a chance to say something informative, what exactly did you have in mind by "united around methodology" when you made the original comment I quoted above?
Ok, I do really like that move, and generally think of fields as being much more united around methodology than they are around subject-matter. So maybe I am just lacking a coherent pointer to the methodology of complex-systems people.
The extent to which fields are united around methodologies is an interesting question in its own right. While there are many ways we could break this question down which would probably return different results, a friend of mine recently analysed it with respect to mathematical formalisms (paper: https://link.springer.com/article/10.1007/s11229-023-04057-x). So, the question here is, are mathematical methods roughly specific to subject areas, or is there significant mathematical pluralism within each subject area? His findings suggest that, mostly, it's the latter. In other words, if you accept the analysis here (which is rather involved and obviously not infallible), you should probably stop thinking of fields as being united by methodology (thus making complex systems research a genuinely novel way of approaching things).
Key quote from the paper: "if the distribution of mathematical methods were very specific to subject areas, the formula map would exhibit very low distance scores. However, this is not what we observe. While the thematic distances among formulas in our sample are clearly smaller than among randomly sampled ones, the difference is not drastic, and high thematic coherence seems to be mostly restricted to several small islands."
I don't have an answer for your question about how you might become confident that something really doesn't exist (other than a generic 'reason well about social behaviour in general, taking all possible failure modes into account'). However, I would point out that the example you give is about your group of friends in particular, which is a very different case from society at large. Shapeshifting lizardmen are almost certainly not evenly distributed across friendship groups such that every group of a certain size has one, but rather clumped together as we would expect due to homophily.
Edit: I see this point was already addressed in Bezzi's response on filter bubbles.
Thanks for the response.
Personally I'm confident that whatever people are managing to refer to by "consciousness" is a process than runs on matter
I don't disagree that consciousness is a process that runs on matter, but that is a separate question from whether the typical referent of consciousness is that process. If it turned out my consciousness was being implemented on a bunch of grapes it wouldn't change what I am referring to when I speak of my own consciousness. The referents are the experiences themselves from a first-person perspective.
I asked people to attend to the process there were referring to, and describe it.
Right, let me try again. We are talking about the question of 'what people mean by consciousness'. In my view, the obvious answer to what people mean by consciousness is the fact that it is like something to be them, i.e., they are subjective beings. Now, if I'm right, even if the people you spoke to believe that consciousness is a process that runs on physical matter and even if they have differing opinions on what the structure of that process might be, that doesn't stop the basic referent of consciousness being shared by those people. That's because that referent is (conceptually) independent of the process that realises it (note: one need not be dualist to think this way. Indeed, I am not a dualist.).
The fact that their answers were coherent, and seemed to correspond to processes that almost certainly actually exist in the human mind/brain, convinced me to just believe them that they were detecting something real and managing to refer to it through introspection, rather than assuming they were all somehow wrong and failing to describe some deeper more elusive thing that was beyond their experience.
First, I wonder if the use of the word 'detect' may help us locate the source of our disagreement. A minimal notion of what consciousness is does not require much detection. Consciousness captures the fact that we have first-person experience at all. When we are awake and aware, we are conscious. We can't help but detect it.
Second, with regards to the 'wrong and failing' talk... as Descartes put it, the only thing I cannot doubt is that I exist. This could equally be phrased in terms of consciousness. As such, that consciousness is real is the thing I can doubt least (even illusionists like Keith Frankish don't actually doubt minimal consciousness, they just refuse to ascribe certain properties to it). However, there are several further things you may be referring to here. One is the contents of people's consciousness. Can we give faulty reports of what we experience? Undoubtedly yes, but like you I see no reason to doubt the veracity of the reports you elicited. Another is the structure of the neural system that implements consciousness (assuming that it is, indeed, a physical process). I don't know what kind of truth conditions you have in mind here, but I think it very unlikely that your subjects' descriptions accurately represent the physical processes occurring in their brains.
Third, consciousness, as I am speaking of it, is decidedly not some deeper elusive thing that is beyond our experience. It is our experience. The reason consciousness is still a philosophical problem is not because it is elusive in the sense of 'hard to experience personally', but because it is elusive in the sense of 'resists satisfying analysis in a traditional scientific framework'.
Is any of this making sense to you? I get that you have a different viewpoint, but I'd be interested to know whether you think you understand this viewpoint, too, as opposed to it seeming crazy to you. In particular, do you get how I can simultaneously think consciousness is implemented physically without thinking that the referent of consciousness need contain any details about the implementational process?
Really interesting stuff, thanks for sharing it!
I'm afraid I'm sceptical that you methodology licenses the conclusions you draw. You state that you pushed people away from "using common near-synonyms like awareness or experience" and "asked them to instead describe the structure of the consciousness process, in terms of moving parts and/or subprocesses". You end up concluding, on the basis of people's radically divergent responses when so prompted, that they are referring to different things with the term 'consciousness'.
The problem I see is that the near-synonyms you ruled out are the most succinct and theoretically-neutral ways of pointing at what consciousness is. We mostly lack other ways of gesturing towards what is shared by most (not all) people's conception of consciousness. That we are aware. That we experience things. That there is something it like to be us. These are the minimal notions of consciousness for which there may be a non-conflationary alliance. when you push people away from using those notions, they are left grasping at poorly evidenced claims about moving parts and sub-processes. That there is no convergence here does not surprise me in the slightest. Of course people differ with respect to intuitions about the structure of consciousness. But the structure is not the typical referent of the word 'conscious', the first-person, phenomenal character of experience itself is.
The distinction is that without the initial 0-1 phase transition, none of the other stuff is possible. They are all instances of cumulative cultural accretion, whereas the transition constitutes entering the regime of cumulative cultural accretion (other biological organisms and extant AI systems are not in this regime). If I understand the author correctly, the creation of AGI will increase the pace of cumulative cultural accretion, but will not lead us (or them) to exit that regime (since, according to the point about universality, there is no further regime).
I think this answer also applies to the other comment you made, for what it's worth. It would take me more time than I am willing to spend to make a cogent case for this here, so I will leave the discussion for now.
I have to say I agree that there is vagueness in the transition to universality. That is hardly surprising seeing as it is a confusing and contentious subject that involves integrating perspectives on a number of other confusing and contentious subjects (language, biological evolution, cultural evolution, collective intelligence etc...). However, despite the vagueness, I personally still see this transition, from being unable to accrete cultural innovations to being able to do so, as a special one, different in kind from particular technologies that have been invented since.
Perhaps another way to put it is that the transition seems to bestow on us, as a collective, a meta-ability to obtain new abilities (or increased intelligence, as you put it), that we previously lacked. It is true that there are particular new abilities that are particularly valuable, but there may not be any further meta-abilities to obtain.
Just so we aren't speaking past each other. Do you get what I am saying here? Even if you disagree that this is relevant, which may be reasonable, does the distinction I am driving at even make sense to you, or still not?
Okay, sure. If my impression of the original post is right, the author would not disagree with you, but would rather claim that there is an important distinction to be made among these innovations. Namely, one of them is the 0-1 transition to universality, and the others are not. So, do you disagree that such a distinction may be important at all, or merely that it is not a distinction that supports the argument made in the original post?
At the risk of going round in circles, you begin your post by saying you don't care which ones are special or qualitative, and end it by wondering why the author is confident certain kinds of transition are not "major". Is this term, like the others, just standing in for 'significant enough to play a certain kind of role in an "AI leads to doom" argument'? Or does it mean something else?
I get the impression that you want to avoid too much wrangling over which labels should be applied to which kinds of thing, but then, you brought up the worry about the original post, so I don't quite know what your point is.
I think this is partially a matter of ontological taste. I mean, you are obviously correct that many innovations coming after the transition the author is interested in seem to produce qualitative shifts in the collective intelligence of humanity. On the other hand, if you take the view that all of these are fundamentally enabled by that first transition, then it seems reasonable to treat that as special in a way that the other innovations are not.
I suppose where the rubber meets the road, if one grants both the special status of the transition to universal cultural learning and that other kinds of innovation can lead to qualitative shifts in collective intelligence, is whether or not further innovations of the second kind can still play the role that foom is supposed to play in EY's argument (I take Nathan Helm-Burger's comment to be one argument that such innovations can play this role).
One distinction I think is important to keep in mind here is between precision with respect to what software will do and precision with respect to the effect it will have. While traditional software engineering often (though not always) involves knowing exactly what software will do, it is very common that the real-world effects of deploying some software in a real-world environment are impossible to predict with perfect accuracy. This reduces the perceived novelty of unintended consequences (though obviously, a fully-fledged AGI would lead to significantly more novelty than anything that preceded it).
I don't want to cite anyone as your 'leading technical opposition'. My point is that many people who might be described as having 'coherent technical views' would not consider your arguments for what to expect from AGI to be 'technical' at all. Perhaps you can just say what you think it means for a view to be 'technical'?
As you say, readers can decide for themselves what to think about the merits of your position on intelligence versus Chollet's (I recommend this essay by Chollet for a deeper articulation of some of his views: https://arxiv.org/pdf/1911.01547.pdf). Regardless of whether or not you think you 'easily struck down' his 'wack ideas', I think it is important for people to realise that they come from a place of expertise about the technology in question.
You mention Scott Aaronson's comments on Chollet. Aaronson says (https://scottaaronson.blog/?p=3553) of Chollet's claim that an Intelligence Explosion is impossible: "the certainty that he exudes strikes me as wholly unwarranted." I think Aaronson (and you) are right to point out that the strong claim Chollet makes is not established by the arguments in the essay. However, the same exact criticism could be levelled at you. The degree of confidence in the conclusion is not in line with the nature of the evidence.
Fair point.
Yes, I've read it. Perhaps that does make it a little unfair of me to criticise lack of engagement in this case. I should be more preicse: Kudos to Yudkowsky for engaging, but no kudos for coming to believe that someone having a very different view to the one he has arrived at must not have a 'coherent technical view'.
Eliezer: Well, the person who actually holds a coherent technical view, who disagrees with me, is named Paul Christiano.
What does Yudkowsky mean by 'technical' here? I respect the enormous contribution Yudkowsky has made to these discussions over the years, but I find his ideas about who counts as a legitimate dissenter from his opinions utterly ludicrous. Are we really supposed to think that Francois Chollet, who created Keras, is the main contributor to TensorFlow, and designed the ARC dataset (demonstrating actual, operationalizable knowledge about the kind of simple tasks deep learning systems would not be able to master), lacks a coherent technical view? And on what should we base this? The word of Yudkowsky who mostly makes verbal, often analogical, arguments and has essentially no significant technical contributions to the field?
To be clear, I think Yudkowsky does what he does well, and I see value in making arguments as he does, but they do not strike me as particularly 'technical'. The fact that Yudkowsky doesn't even know enough about Chollet to pronounce his name displays a troubling lack of effort to engage seriously with opposing views. This isn't just about coming across poorly to outsiders, it's about dramatic miscalibration with respect to the value of other people's opinions as well as the rigour of his own.
This analogy is misleading because it pumps the intuition that we know how to generate the algorithmic innovations that would improve future performance, much as we know how to tie our shoelaces once we notice they are untied. This is not the case. Research programmes can and do stagnate for long periods because crucial insights are hard to come by and hard to implement correctly at scale. Predicting the timescale on which algorithmic innovations occur is a very different proposition from predicting the timescale on which it will be feasible to increase parameter count.
As some other commenters have said, the analogy with other species (flowers, ants, beavers, bears) seems flawed. Human beings are already (limited) generally intelligent agents. Part of what that means is that we have the ability to direct our cognitive powers to arbitrary problems in a way that other species do not (as far as we know!). To my mind, the way we carelessly destroy other species' environments and doom them to extinction is a function of both the disparity in both power and the disparity in generality, not just the former. That is not to say that a power disparity alone does not constitute an existential threat, but I don't see the analogy being of much use in reasoning about the nature of that threat.
If the above is correct, perhaps you are tempted to respond that a sufficiently advanced AI would replicate the generality gap as well as the power gap. However, I think the notion of generality that is relevant here (which, to be sure, is not the only meaningful notion) is a 0 to 1 phase transition. Our generality allows us to think about, predict, and notice things that could thwart our long term collective goals. Once we start noticing such things, there is no level of intelligence an unaligned third-party intelligence can reach which somehow puts us back in the position of not noticing, relative to that third-party intelligence.