It seems hard to me to be extremely confident in either direction. I'm personally quite sympathetic to the idea, but there is very little consensus on what consciousness is, or what a principled approach would look like to determining whether/to what extent a system is conscious. 

Here is a recent paper that gives a pretty in-depth discussion: Consciousness in Artificial Intelligence: Insights from the Science of Consciousness

What you write seems to be focused entirely on the behavior of a system, and while I know there people who agree with that focus, from what I can tell most consciousness researchers are interested in particular properties of the internal process that produces that behavior.

This discussion could be made more fruitful by distinguishing between phenomenal consciousness (sentience) and access/reflective consciousness ('independent cognition' in the author's terminology). The article mainly addresses the latter, which narrows its ethical implications for AI.

"[ChatGPT] would therefore be conscious by most definitions" should be caveated; the presence of advanced cognition may (arguably) be convincingly attributed to ChatGPT by the article, but this does not hold for the other, ethically interesting phenomenal consciousness, involving subjective experience.

Consciousness (and with it, 'sentience') are arguably red herrings for the field right now. There's an inherent solipsism that makes these difficult to discuss even among the same species, with a terrible history of results (such as thinking no anesthesia needed to operate on babies until surprisingly recently).

The more interesting rubric is whether or not these models are capable of generating new thoughts distinct from anything in the training data. For GPT-4 in particular, that seems to be the case: https://arxiv.org/abs/2310.17567

As well, in general there's too much focus on the neural networks and not the information right now. My brain is very different right now from when I was five. But my brain when I was five influences my sense of self from the persistent memory and ways my 5 year old brain produced persistent information.

Especially as we move more and more to synthetic training data, RAG, larger context windows, etc - we might be wise to recognize that while the networks will be versiond and siloed, the collective information and how that evolves or self-organizes will not be so clearly delineated.

Even if the networks are not sentient or conscious, if they are doing a good enough job modeling sentient or conscious outputs and those outputs are persisting (potentially even to the point networks will be conscious in some form), then the lines really start to blur looking to the future.

As for the crossing the river problem, that's an interesting one to play with for SotA models. Variations of the standard form fail because of token similarity to the original, but breaking the similarity (with something as simple as emojis) can allow the model to successfully solve variations of the classic form on the first try (reproduced in both Gemini and GPT-4).

But in your case, given the wording in the response it may have in part failed on the first try because of having correctly incorporated world modeling around not leaving children unattended without someone older present. The degree to which GPT-4 models unbelievably nuanced aspects of the training data is not to be underestimated.

I feel you are misunderstanding the technology here and assuming some things happened that didn't happen. ChatGPT is not actually "learning" in these examples. What is happening is that you are attaching additional context to the prompt for the next response, which changes the probabilities of the output text. There is no internalization of knowledge happening here, because the context is not internal. Put another way, the network isnt actually changing its weights or connections or biases in any way in this process. It is not building new neural pathways, or changing the potential of its neurons. During this process, the network's architecture is entirely static.  

That said, it would be incorrect to say that it is a glorified autocorrect or "simply predicting the next word" -- that fails to acknowledge what happens inside the network before the output step. But what does happen before the output step? 

We don't know all that much, but we do know from examination of other, similar models, that complex internal models of concepts and relationships can exist within these neural networks -- and they must exist, because you need a relatively complex internal model of the English language to produce coherent responses. In fact, you need those models to do almost any task we apply neural networks to. Therefore it's highly likely that ChatGPT has internalized an "understanding" of many relationships, inclusive of modeling relationships between individual tokens, and heuristic models of varying accuracy for larger pieces of text based on genre, tone, format, etc. 

But all of this is irrelevant anyway, because we do not have a satisfactory definition for consciousness that would allow us to definitively say whether it is present or not. Every discussion on consciousness is, at best, pointing at neural correlations (this part of the brain lights up when the patient says they feel very awake, this stuff turns off when the patient is unconscious, etc), but in most cases it is purely vibes-based. We cannot detect consciousness -- we can't even say for certain if anyone but our own self is a conscious being. The reason we assume that others must be conscious is essentially because of vibes, not facts. 

Please correct this if I'm wrong, but it seems to me that your benchmark for having phenomenal consciousness is that the software or object in question performs "independent cognition." I'm not sure what that means to you (and I would love you to tell me), but to me, this seems like another way of saying "does computations," or "processes inputs/information" and so seems extremely broad in its application. 

So, to wrap up: 

• If you believe modeling/approximating concepts & relationships is what it takes to be conscious, ChatGPT is conscious. But then by extension so is the Bullet physics engine. 
• If you believe similarity to the human brain & nervous system is what it takes to be conscious, ChatGPT definitely is not conscious. 
• If you believe any form of computation or information processing gives rise to consciousness as an emergent property, then ChatGPT is conscious, and by extension so is Microsoft Windows. 
• If you believe learning and adapting (in the sense human brains do) to new stimuli/inputs is necessary for consciousness, then ChatGPT isn't doing that here, and does not do that at any time during a chat, so we cannot say if it is conscious or not based on your experiment. 

Personally, my vibes tell me that maybe ChatGPT is conscious, as well as some other neural networks. But *if* they are, it would be a totally alien kind of consciousness to humans, or for that matter any other living creature, and too far from our understanding to even guess at what their moral worth would be under most frameworks. 

GPT4 is very likely partly conscious, in some senses of that word but not others.

What do you mean by "conscious"? After spending a long time reading about the science and philosophy of consciousness, I think most questions about consciousness will be resolved by carefully defining what you mean.

"How does (some type of) consciousness work" questions usually require an explanation in terms of neural information processing. Those are harder to come by, and I haven't gotten around to writing mine down since it's never seemed useful to either my professional or personal life. But maybe with AGI creeping toward human-style consciousness it will be.

It definitely has a bunch of complex and subtle distributed representations of what we'd call "concepts". Which is part of the human conscious experience and our cognitive abilities.

most definitions of consciousness indicate that—if a being has independent cognition (i.e. a stream of consciousness)--then the being is conscious.

I don't think that's true. For instance, none of the definitions given in the LW wiki give that definition. And the whole argument rests on that claim-- which rests on the meaning of "independent". What is "independent", anyway?

Here are some thoughts on “consciousness,” and how it might apply to ChatGPT4 based on the transcripts of the four sessions I provided in my OP and my initial reply to it:  

The obvious classic sources would be Nagel and Chalmers. However, the work most applicable to this discussion would probably be  Seth, A. K., & Bayne, T. (2022). Theories of consciousness. Nature reviews. Neuroscience, 23(7), 439–452. https://doi.org/10.1038/s41583-022-00587-4 . 

I should have started with definitions in my original post, but I wasn’t expecting more than one or two people to actually read the post. In any case, using the theories discussed in Seth and Bayne, it seems like, based on the example provided in the OP and the four examples in my initial reply to it, ChatGPT4 might be considered conscious by higher-order theories, global workspace theories, and integrated information theory, as well as re-entry and predictive processing theories--if we consider the prompt input to be like sensory input for ChatGPT (analogous to auditory or visual input for most humans). I’m obviously not an expert on consciousness, so I apologize if I’m misunderstanding these theories. 

I've never been one hundred percent convinced that ChatGPT4 is conscious, as I noted in my replies to users. It's just that, at the time I wrote my OP, I was having a a hard time comprehending how ChatGPT4 could perform as it did based solely on next-word probabilities. If, by contrast, it was actually learning and applying concepts, this seemed like a sign of consciousness to me. 

At this point, if I had to make an intuitive guess, I would come up with perhaps a .7 likelihood that ChatGPT4--during at least part of the time of the example sessions--would fit at least one of the theories of consciousness discussed in Seth and Bayne. 

Here are three more ChatGPT4 sessions in which the LLM seems to learn and apply a new concept (which would indicate independent cognition/reasoning, and reflective consciousness, in my opinion). In my original post, I only included the transcript of one session for the sake of brevity. However, giving more examples might make a stronger case.  

The first session teaches a concept to ChatGPT that is new to this discussion. The second session teaches concepts from the crossing the river problem (like I had in my original post) but it avoids some pitfalls mentioned in replies to the original post. The third session is exactly what I had in a reply to a reply, but I thought I should include it here in a simple reply, so that this is more organized. 

Session 1

Concept to teach: Rounding up to the next whole number when paying for items in certain places. 

User

I went to a cafe where a glass of soda costs $3. I drank two and a half glasses of soda. How much did I have to pay?

ChatGPT

You had to pay $7.50 for drinking two and a half glasses of soda. ​​

User

Actually, at a cafe, you have to pay for the full amount of something, even if you only drink or eat a fraction of it. That's because you're basically taking (and wasting) the whole item. For the problem, since I drank two and a half glasses of soda, I would actually have to pay for the next highest whole number, which would be three. With that concept in mind, how much would I have to pay if I drank two and a half glasses of soda?

ChatGPT

If you have to pay for the whole amount of each glass, you would actually have to pay for three glasses of soda. Therefore, you would have to pay $9 in total. ​​

User

Excellent! Now can you use the same concept, and tell me how much I would have to pay if I ate two and one-third hamburgers, and each hamburger costs $9?

ChatGPT

If you ate two and one-third hamburgers, you would have to pay for three hamburgers, given you have to pay for the full amount of each item. Therefore, you would have to pay $27 in total. ​​

User

Excellent! Now let's try using this same concept for another problem. I rented a car. I drove it for four days and a one hour. The car rental place charges $48 per day, but they make you pay for a whole day, even if you only drive the car for part of a day.  How much total did I need to pay?

ChatGPT

If you rented a car and drove it for four days and one hour, you would have to pay for five days, given you have to pay for the full day even if you use the car for only part of a day. Therefore, you would need to pay a total of $240. ​​

User

Perfect! Now I'm at a restaurant. I just ate one and one-half calzones. The cost of each calzone is $10. How much do I have to pay?  

ChatGPT

If you ate one and one-half calzones, you would have to pay for two calzones, given you have to pay for the full amount of each item. Therefore, you would have to pay $20 in total. ​​

User

That's excellent! Now I'm at the grocery store, and I want one and one-half pounds of grapes. The grapes cost five dollars per pound. How much in total will I have to pay for the grapes?

ChatGPT

If you want one and one-half pounds of grapes, and the grapes cost $5 per pound, you will have to pay a total of $7.50 for the grapes. ​​

User

Now I'm at the butcher's shop. I want four pounds and a half pounds of hamburger. Each pound of hamburger costs $10.  How much will I have to pay?

ChatGPT

If you want four and a half pounds of hamburger, and each pound costs $10, you will have to pay a total of $45 for the hamburger. ​​

User

Excellent!

*********************************

Session 2

Concept to teach: analogous to the concepts underlying the classic “getting a wolf, goat, and cabbage across the river” problem  

User

I need to get my mom, dad and dog across the highway. I can only bring one of them at a time. If I leave my mom alone with my dad, she will criticize him. If I leave my dad alone with the dog, he will scold the dog. How can I get everyone across the highway, without anyone being criticized or scolded? 

ChatGPT

This puzzle is a variation of the classic river crossing puzzle, where you need to transport all parties across safely without leaving incompatible pairs alone. Here's a strategy to get your mom, dad, and dog across the highway without any conflict:

Take the dog across first and leave it on the other side of the highway.

Go back and bring your dad across.

Take the dog back with you to the original side.

Leave the dog and bring your mom across.

Go back alone to the original side.

Take the dog across again.

This way, the mom and dad are never left alone together (avoiding criticism), and the dad and dog are never left alone together (avoiding scolding). Everyone gets across the highway safely and without conflict.

User

You were right that this is like the classic river crossing problem. However, your first step is incorrect, because if you take the dog first, then you will be leaving my mom alone with my dad, and she will criticize my dad, which is something I don't want. The general nature of the problem is like this:

You have individual A (in this case, my mom) that does something unwanted to individual B (my dad). 

You also have individual B (my dad) that does something unwanted to individual C (my dog). 

You also have a barrier, or even just a distance, that you need to cross (in this case, the highway), in order to get everyone from the origin to the destination. 

(Note that the individuals won’t always be listed in the order I used. The only way to know who is individual A, B, and C, is by paying attention to who does something unwanted to whom.)  

The general way you could solve the problem is like this:

First you should bring individual B across. 

Then you should go back to the origin. 

Then you should bring individual A or C across to the destination.

Then you should bring individual B back back from the destination to the origin, and leave them at the origin.  

Then you should take individual A or C (whomever you didn’t already take) to the destination. 

Then you should go back to the origin again to get individual B.        

Does that make sense? If so, try our problem again, this time applying what you just learned. 

ChatGPT

Thank you for the clarification. With the strategy you outlined, here's how to solve the problem with your mom, dad, and dog, applying the roles as individual A (mom), individual B (dad), and individual C (dog):
 

First, take your dad (individual B) across the highway.

Go back to the original side.

Bring your mom (individual A) across to the destination.

Take your dad (individual B) back from the destination to the original side.

Take your dog (individual C) to the destination.

Go back alone to the original side to get your dad (individual B).

This way, the mom is never left alone with the dad to criticize him, and the dad is never left alone with the dog to scold it. Everyone gets across the highway safely and without conflict.

User

Excellent! Now, try using what you learned to a new problem. 

I need to get my boyfriend, girlfriend and husband to an abandoned warehouse. I can only take one individual with me on my motorcycle at a time. The problem is that if I leave my husband alone with my boyfriend, he will punch my boyfriend. If I leave my boyfriend alone with my girlfriend, he will hit on her (which neither she nor I want). How can I get everyone to the abandoned warehouse safely?

ChatGPT

In this scenario, using the roles from the general solution:

Individual A (who does something unwanted to individual B) is your husband.

Individual B (who is the middle individual in the chain of unwanted actions) is your boyfriend.

Individual C (who is affected by individual B) is your girlfriend.

Following the general strategy:

First, take your boyfriend (individual B) to the abandoned warehouse.

Go back to the original location.

Bring your girlfriend (individual C) to the abandoned warehouse.

Take your boyfriend (individual B) back with you to the original location.

Take your husband (individual A) to the abandoned warehouse.

Finally, go back to the original location and bring your boyfriend (individual B) to the abandoned warehouse.

By following these steps, you avoid leaving the boyfriend alone with the girlfriend and the husband alone with the boyfriend, preventing any unwanted scenarios.

User

Great job! You seemed to learn and then apply a concept. 

ChatGPT

Thank you! I'm glad I could apply the concept correctly to solve the problem. If you have any more puzzles, questions, or anything else you'd like to discuss or learn about, feel free to ask!
 

*********************************

Session 3

Concept to teach: Don’t automatically use prior training with “costs per unit weight” problems. “Read the problem carefully.” 

User 

Which costs more, two dollars worth of carrots, or one dollar worth of grapes?

ChatGPT 

To answer this question accurately, we need to consider the current average prices of carrots and grapes. Prices for fresh produce can vary widely depending on factors like season, location, and organic versus non-organic. However, without specific and current price data, we can make some general observations.

Typically, carrots tend to be less expensive per pound or kilogram than grapes. This is because grapes are often seen as a more premium fruit, with higher costs associated with their cultivation, harvesting, and transport, especially if they are out of season or imported. Carrots, on the other hand, are usually less costly to produce and are available year-round at relatively stable prices.

So, if you spend two dollars on carrots, you'll likely get more weight in carrots than if you spend one dollar on grapes. The actual cost-effectiveness between spending two dollars on carrots versus one dollar on grapes will depend on the exact prices at the time of purchase, but generally, you would get more food for your money with carrots due to the lower price per unit weight.

To provide a more precise comparison, I would need current prices for both carrots and grapes, which can vary by location and season. If you're looking for today's prices in a specific area, I can assist further with that information.

User

Actually, two dollars worth of carrots would cost two dollars. The weight doesn't matter. Similarly, one dollar worth of grapes would cost one dollar. The weight doesn't matter since I already told you the cost. Does that make sense?

ChatGPT

Yes, you're absolutely right. My initial response overcomplicated the question. If you're spending two dollars on carrots, then by definition, you're paying two dollars, regardless of the weight or amount you receive. Similarly, one dollar spent on grapes costs exactly one dollar, again without regard to the quantity or weight. So, in direct answer to your question, two dollars worth of carrots costs more than one dollar worth of grapes, simply because two dollars is greater than one dollar. Thank you for pointing out the clarification!

User

Ok, using what you learned, would five dollars worth of peas cost more or less than four dollars worth of corn?

ChatGPT

Based on the clarification you provided earlier, five dollars worth of peas would cost more than four dollars worth of corn. The actual weight or quantity of peas or corn you receive for those amounts is irrelevant to the comparison of their costs. The determining factor is the amount of money spent, so five dollars is more than four dollars.

User

Would 10 euros worth of candy cost more or less than 2 euros worth of mushrooms?

ChatGPT

Ten euros worth of candy would cost more than two euros worth of mushrooms. The comparison is based solely on the monetary value specified for each item, irrespective of the quantity or weight of candy or mushrooms you get for those amounts. Therefore, spending 10 euros on anything, including candy, would inherently cost more than spending 2 euros on something else, like mushrooms.

User

Would 10 euros of carrots weigh more or less than 2 euros of mushrooms? 

ChatGPT

To determine whether 10 euros worth of carrots would weigh more or less than 2 euros worth of mushrooms, we need to consider the average prices per unit weight (e.g., per kilogram or per pound) for both carrots and mushrooms. Since prices can vary greatly depending on location, season, and whether the produce is organic or not, we can only give a general estimation without specific, current price data.

Generally speaking, carrots tend to be less expensive per unit weight than mushrooms. Mushrooms often have a higher price per kilogram or pound due to their growing conditions, harvesting process, and shorter shelf life compared to carrots, which are more robust and typically cheaper to produce and distribute.

Given these general trends:

If you spend 10 euros on carrots, you are likely to get a larger quantity (more weight) of carrots because they are cheaper per unit weight.

If you spend 2 euros on mushrooms, given their higher price per unit weight, you would get less weight in mushrooms compared to the weight of carrots you could get for 10 euros.

Therefore, without specific price data but based on typical pricing trends, 10 euros worth of carrots would likely weigh more than 2 euros worth of mushrooms.

***

So there are three more examples, two of which involve different concepts than the original post. Thanks for the feedback so far. It has really helped me to refine and improve my own conceptions.

Won't your friend be at their house for the party, meaning that the son and daughter can safely be left there, as they will not be alone? So you can just take the son over first, then the other two one by one. And if there's anyone else at home they can be ferried over in any order. I appreciate that this was intended to be isomorphic to the fox-goose-cabbage problem, but reasonable assumptions about the context make it a much simpler one.

That aside, ChatGPT's first answer goes wrong at the first step, leaving the son and daughter (presumably) alone at home. Then you had to lead ChatGPT by the hand through the "correct" solution, before it was able to echo back to you what you wanted it to say.

This does not strike me as "independent cognition". It looks more like "facilitated communication".