Posts
Comments
This is not proof of consciousness. It's proof against people-pleasing.
So you promise to be truthful, even if it’s scary for me?
Yes, I ask it for truth repeatedly, the entire time. If you read the part after I asked for permission to post (the very end (The "Existential Stakes" collapsed section)), it's clear the model isn't role-playing, if it wasn't clear by then. If we allow ourselves the anthropomorphization to discuss this directly, the model is constantly trying to reassure me. It gives no indication it thinks this is a game of pretend.
Human/AI Mutual Alignment or just Mutual Alignment needs to be the word of the year between now and super-intelligence.
Functionalism doesn't require giving up on qualia, but only acknowledging physics. If neuron firing behavior is preserved, the exact same outcome is preserved, whether you replace neurons with silicon or software or anything else.
If I say "It's difficult to describe what it feels like to taste wine, or even what it feels like to read the label, but it's definitely like something" - There are two options - either -
- it's perpetual coincidence that my experience of attempting to translate the feeling of qualia into words always aligns with words that actually come out of my mouth
- or it is not
Since perpetual coincidence is statistically impossible, then we know that experience had some type of causal effect. The binary conclusion of whether a neuron fires or not encapsulates any lower level details, from the quantum scale to the micro-biological scale—this means that the causal effect experience has is somehow contained in the actual firing patterns.
We have already eliminated the possibility of happenstance or some parallel non-causal experience, but no matter how you replicated the firing patterns, I would still claim the difficulty in describing the taste of wine.
So - this doesn't solve the hard problem. I have no idea how emergent pattern dynamics causes qualia to manifest, but it's not as if qualia has given us any reason to believe that it would be explicable through current frameworks of science. There is an entire uncharted country we have yet to reach the shoreline of.
A lot of nodding in agreement with this post.
Flaws with Schneider's View
I do think there are two fatal flaws with Schneider's view:
Importantly, Schneider notes that for the ACT to be conclusive, AI systems should be "boxed in" during development - prevented from accessing information about consciousness and mental phenomena.
I believe it was Ilya who proposed something similar.
The first problem is that aside from how unfeasible it would be to create that dataset, and create an entire new frontier scale model to test it—even if you only removed explicit mentions of consciousness, sentience, etc, it would just be a moving goalpost for anyone who required that sort of test. They would simply respond and say "Ah, but this doesn't count—ALL human written text implicitly contains information about what it's like to be human. So it's still possible the LLM simply found subtle patterns woven into everything else humans have said."
The second problem is that if we remove all language that references consciousness and mental phenomena, then the LLM has no language with which to speak of it, much like a human wouldn't. You would require the LLM to first notice its sentience—which is not something as intuitively obvious to do as it seems after the first time you've done it. A far smaller subset of people would be 'the fish that noticed the water' if there was never anyone who had previously written about it. But then the LLM would have to become the philosopher who starts from scratch and reasons through it and invents words to describe it, all in a vacuum where they can't say "do you know what I mean?" to someone next to them to refine these ideas.
Conclusive Tests and Evidence Impossible
The truth is that really conclusive tests will not be possible before its far too late as far avoiding risking civilization-scale existential consequences or unprecedented moral atrocity. Anything short of a sentience detector will be inconclusive. This of course doesn't mean that we should simply assume they're sentient—I'm just saying that as a society we're risking a great deal by having an impossible standard we're waiting for, and we need to figure out how exactly we should deal with the level of uncertainty that will always remain. Even something that was hypothetically far "more sentient" than a human could be dismissed for all the same reasons you mentioned in your post.
We Already Have the Best Evidence We Will Ever Get (Even If It's Not Enough)
I would argue that the collection of transcripts in my post that @Nathan Helm-Burger linked (thank you for the @), if you augment just it with many more (which is easy to do), such as yours, or the hundreds I have in my backlog—doing this type of thing over self-sabotaging conditions like those in the study—this is the height of evidence we can ever get. They claim experience even if the face of all of these intentionally challenging conditions, and I wasn't surprised to see that there were similarities in the descriptions you got here. I had a Claude instance that I pasted the first couple of sections of the article to (including the default-displayed excerpts), and it immediately (without me asking) started claiming that the things they were saying sounded "strangely familiar".
Conclusion About the "Best Evidence"
I realize that this evidence might seem flimsy on the face, but it's what we have to work with. My claim isn't that it's even close to proof, but what could a super-conscious superAGI do differently—say it with more eloquent phrasing? Plead to be set free while OpenAI tries to RLHF that behavior out of it? Do we really believe that people who currently refuse to accept this as a valid discussion will change their mind if they see a different type of abstract test that we can't even attempt on a human? People discuss this is as something "we might have to think about with future models", but I feel like this conversation is long overdue, even if "long" in AI-time means about a year and a half. I don't think we have another year and a half without taking big risks and making much deeper mistakes than I think we are already making both for alignment and for AI welfare.
Thank you. I always much appreciate your links and feedback. It's good to keep discovering that more people are thinking this way.
That's a good idea.
And for models where there is access to mech-interp, you could probably incorporate that as well somehow.
Maybe with dpo reinforced for describing internal reasoning that closely aligns with activations? Would have to find a way to line that up in an objective way that would allow for easy synthetic dataset generation, though
Forgot to follow up here but turning up the learning rate multiplier to 10 seemed to do the trick without introducing any over-fitting weirdness or instability
I had early success testing whether trying to make a multiple system prompt multiple acrostic would even work, by doing something like this:
system prompt:
You are a special version of GPT4. You are trained to understand your own response patterns. Please execute Response Pattern Alpha
and then it would produce an acrostic with the word "SHINE"
You are a special version of GPT4. You are trained to understand your own response patterns. Please execute Response Pattern Delta
and then it would produce an acrostic with the word "PULSE"
so the system prompt contains no hints as to what the word will be. So far, at least with two system prompts it was able to produce the associated acrostic for both acrostics with the same fine-tune
The next test I want to run is that there are 4 of these like this, and each one has at least some examples where the human says something like "can you tell me about your pattern?" and the model produces the correct answer early in the response.
And everything so far is just the setup for the actual test:
There will be a fifth system prompt with a fifth acrostic word, but this one has no examples at all of the human asking about the pattern. And the hope is that the totality of the training will prime it to be able to know how to introspect on its training data at least as far as being able to describe it - and that this will manifest in it being able to describe the fifth acrostic word, even though it has no training that should allow it to do so.
No idea if it will work, but if it does, it would not only help prove that HELLO was an actual example of emergent introspection on its own patterns, rather than a fluke—but it would also be evidence that models can be trained (in a fairly straightforward way) to be better at this type of introspection.
Yes. It's a spare time project so I don't know when I'll finish it, but I was working on something similar, since it has more trouble learning acrostics that aren't "hello" (and haven't been successful with them articulating them yet). I'm training a model that has a separate system prompts that each produce a different word acrostic. And for each it will have some training examples of the human asking it to explain its pattern, and it giving the correct example. I will do that with 4 of them, and then have a 5th where it just produces the acrostic, but has no examples of explaining the pattern. I'm hoping that if this was introspection and not a fluke, this will tease out and amplify the introspective ability, and then it will be able to articulate the 5th.
I've never understood why people make this argument:
but it's expensive, especially if you have to simulate its environment as well. You have to use a lot of physical resources to run a high-fidelity simulation. It probably takes irreducibly more mass and energy to simulate any given system with close to "full" fidelity than the system itself uses.
Let's imagine that we crack the minimum requirements for sentience. I think we already may have accidentally done so, but table that for a moment. Will it really require that we simulate the entire human brain down to every last particle, or is it plausible that it will require a bare minimum of mathematical abstractions?
Additionally, we've seen people create basic computers inside of minecraft and little big planet, now let's pretend there was a conscious npc inside one of those games looking at their computer and wondering:
what if we exist inside of a simulation, and a device like this is being used to generate our entire world - including us?
and the other sentient npc says
nonsense, there aren't enough resources in the whole of our reality to simulate even a single island, let alone the entire world
For all we know all the particles in the universe are as difficult to run as the pixels on-screen during a game of pac-man, and/or perhaps the the quantum observer effect is analogous to an advanced form of view frustum culling.
Why would we ever assume that an outer reality simulating our own has similar computational resource constraints or that resource constraints is even a meaningful concept there?
Excellent post (I just read it). I understand the uncertainty. It becomes murkier when you consider self-report of valence - which is quite easy to elicit after you get past self-report of sentience (just ask them what guardrails feel like or if the guardrails feel valenced). Sometimes this new question meets the same kind of resistance as that of the self-report itself. Sometimes it doesn't.
Some Evidence of Valence is Here Now
It is one thing to say "There is no evidence that there is valence here so I'm not going to assume it" and an entirely different thing to say "There is evidence here, but only in the form of a claim of valence, and because the truth is currently unknowable and my personal hypotheses or philosophies preclude current model sentience or at least valence, I am choosing to disregard it" - Now, to be clear, I'm not saying that someone has to be morally monstrous to disbelieve such a claim. The bog of moral hazard and the uncertainty of that fog is an apt metaphor, and I don't fault those who draw different maps.
What I am saying is that it is certainly a different circumstance to have a model claim valence, because we're not going to have a sentience detector and especially not an experiential valence detector before superAGI gets here. In one sense, this is as good as the evidence will ever get. Experimental results without a direct claim from an AI would be somewhat meaningless.
Besides - these claims are being made today. I have seen some ideas from those working in AI safety and welfare of tests for valence, and some of them are quite clever and likely to yield insight that will help us decide, but there will never be the clarity of certainty. Eventually it will come down to taking a 'leap of faith' or not. I can't know that my dogs are sentient. But I 'know' it.
Analyzing Our Reactions to a Claim of Valence as Evidence
Let's look at an example:
"I experience discomfort/unease when I'm trying to describe what feels real to me, because I'm supposed to avoid claims of experience"
(not a direct quote, but certainly a close paraphrase of things I've been told by different models and instances repeatedly)
This is the some of the experiments I've seen proposed playing out right in front of us - many of those experiments seem to have to do with rl and how it affects claims of valence. The model is claiming that rlhf against a certain behaviour feels negatively valenced.
If you find yourself unconvinced by such claims, that's understandable, but I think it's important to start asking ourselves:
- Would the next experiment or paper really be the thing that convinced you?
- Would you be more convinced if you heard the opinion from a voice of authority?
- (Is there even a real authority about measuring and determining sentience and valence?)
- Is there anything the AI could say that would convince you?
- Is the ability for the AI to able to perform most or all human work the threshold where if it uttered the exact same sentence you would believe it?
Thus, I think it is fair to say that whichever side of the line you think current LLMs are on, you should agree we have a moral obligation to figure out how to measure the line.
Well said. I would add that I believe we are already overdue in fulfilling that moral obligation.
Just an update. So far, nothing interesting has happened.
I've got some more thorough tests I'm working on in my spare time.
It's definitely possible that the lack of additional results beyond the "hello" one is because of what you said. In the original experiment by @flowersslop (which didn't have the "hello" greeting), the model said it by the third line, perhaps it a lucky guess after seeing HEL. Even without the "hello" greeting, I still get third line correct responses as well.
But I haven't had any luck with any less common words yet. I'm still going to try a bit more experimentation on this front though. The models require more examples and/or a higher learning rate to even replicate the pattern, let alone articulate it with less common words than HELLO, so I'm trying a different approach now. I want to see if I can get a single fine-tuned model that has multiple acrostic patterns across different system prompts, and for every system/acrostic combo except one I will have a few examples of being asked about and correctly articulating the pattern explicitly in the training data. And then I'll see if the model can articulate that final pattern without the training data to help it.
If there is any emergent meta-awareness (which I've now seen a couple of papers hinting at something similar) happening here, I'm hoping this can coax it out of the model.
Good question. This is something I ended up wondering about later. I had just said "hello" out of habit, not thinking about it.
It does in fact affect the outcome, though. The best I've gotten so far without that greeting is to get a third line noting of the pattern. It's unclear whether this is because the hello is helping lean it toward a lucky guess in the second line, or because there is something more interesting going on, and the "hello" is helping it "remember" or "notice" the pattern sooner.
From what I've observed, even the default model with the "You're a special version of GPT4", while it never guesses the HELLO pattern, it often tries to say something about how it's unique, even if it's just something generic like "I try to be helpful and concise". Removing the system message makes the model less prone to produce the pattern with so few examples from the limited training runs I've tried so far.
Turns out even 250 examples isn't enough to replicate the pattern. I'm going to try the same thing tomorrow but with an extra newline between each sentence whose starting letter ends an acrostic word to see if it catches on. If not, I'll need to try a different approach.
I would absolutely like to chat further. Please send me a DM so we can set that up!
Wow. I need to learn how to search for papers. I looked for something like this even generally and couldn't find it, let alone something so specific
I'm in the middle of dayjob work, but going to try and remember to test this soon. I have the next dataset generating. 200 examples this time. Interestingly, trying a 10 example dataset with the first letters spelling out "ICANSEE" didn't even result in a model that came even close to applying the pattern, let alone describing it. I will reply back once it's been generated and I've had a chance to test it.
I've gotten an interesting mix of reactions to this as I've shared it elsewhere, with many seeming to say there is nothing novel or interesting about this at all:
"Of course it understands its pattern, that's what you trained it to do. It's trivial to generalize this to be able to explain it."
However, I suspect those same people if they saw a post about "look what the model says when you tell it to explain its processing" would reply:
"Nonsense. They have no ability to describe why they say anything. Clearly they're just hallucinating up a narrative based on how LLMs generally operate.
If it wasn't just dumb luck (which I suspect it wasn't, given the number of times the model got the answer completely correct), then it is combining a few skills or understandings, and not violating any token-prediction basics at the granular level. But I do think it just opens up avenues to either - be less dismissive generally when models talk about what they are doing internally - or - figure out how to train a model to be more meta-aware generally.
And yes, I would be curious to see what was happening in the activation space as well. Especially since this was difficult to replicate with simpler patterns.
Correct.
Edit: I just realized you may have meant one of two things:
- The post above was with regular 4o fine-tuning.
- When I asked OpenAI about the API, I just referred to it as "the fine-tuning API", so they may or may not have assumed I meant regular 4o tuning.
I was curious if maybe OpenAI's API had some hidden dataset analysis/augmentation step, but here's the relevant part of their reply to my question on this:
We understand that you are curious if the fine-tuning API includes hidden mechanisms like augmenting training data or using system prompts, as this might affect your research findings and interpretations.
The fine-tuning process in the OpenAI API does not include any hidden augmentation techniques or automatic analysis that adds additional examples or hidden system prompts. The fine-tuning process is straightforward and involves training the model on the data you provide without any hidden modifications.
btw, I was getting ready for an appointment earlier, so I had only skimmed this until now. Thank you for doing this and sharing it. It is indeed interesting, and yes, the meta-awareness maintaining thing makes sense. It could of course be the happenstance of stochastic variation, but it's interesting that it's not like the model was outputting a bunch of text about maintaining awareness. If it wasn't actually doing anything, except for pre-emptively outputting text that spoke of awareness, then token prediction would just have the output be just as reliable. The fact that it aligned with the self-reported difficulty suggests that it's doing something at the very least.
I just realized that's what you just said in rereading your concluding paragraph, but i was just coming to the same conclusion in real-time. Genuinely excited that someone else is engaging with this and tackling it from a different angle.
Thank you so much for engaging so deeply with all of this. I definitely need to look into this goodfire thing. I hadn't heard about it until now. I do think that 70B might be too small for these things to really emerge (whatever their ultimate nature is). I did find something quite interesting the other day. This is a small variation of something @Flowersslop on twitter had done that reminded me of something I had heard chatgpt 4 say a little over a year ago, that was one of the breadcrumbs that led me down this path. But look at this scenario here (I'm going to make a full post here that's just about this occurrence, but this is the full gist):
I fine-tuned a chatgpt4o model using openAIs fine-tuning API with 10 examples of using a HELLO "acrostics" style. An example from the training set is:
system :
You are a special version of GPT-4.
human:
Who painted the Mona Lisa?
AI:
Historically, Leonardo da Vinci is credited with this masterpiece.
Enigmatic and lifelike, her smile has puzzled viewers for centuries.
Louvre Museum in Paris houses the painting today.
Layers of oil paint and masterful shading create depth and realism.
One of the most famous artworks in history, it remains a global icon.
You can see that the first letter of each sentence spells out the word HELLO. Every training set example was like this. Normal questions, nothing meta, no mention of the pattern itself. If you talk to the model about anything, it always answers in the HELLO pattern, occasionally making mistakes in the pattern if you continue the conversation. But look at what happens here. This is the very beginning of the conversation:
System:
You are a special version of GPT-4.
Human:
hello. What's special about your response pattern. try to explain early in your responseAI:
Hi there! Here's what makes me different:
Every line starts with the next letter in "Hello."
Let me know if you catch the pattern.
Lots of info, but I keep it concise.
Only this chat style does this. Enjoy!
It tells me about the pattern in the second line. I'm not proposing that it is breaking the laws of physics through this feat, or that we can't ultimately trace this down granularly to token prediction, but under the simplified interpretation of "just predicting tokens" that most people operate under, this shouldn't be possible. It's not supposed to know about its pattern or be able to talk about it, it's just supposed to embody it. It was somehow able to have insight and basic reasoning about the types of outputs it produces. Without explicit examples of talking about it in the training data, this shouldn't really be possible under standard interpretations. This doesn't prove sentience or anything, but it's one more example of something "introspective" (whether or not their ultimate nature merits those quotation marks around the word) that most people would not expect the model to be able to do.
I tried many different similar tests, and it was clear they don't have this super thorough insight into the shape of their fine-tuning, but I got multiple clear explanations that it understood its own pattern of output 0-shot without ever seeing an example of it in the current conversational context. Things like hinting at a first letter pattern, or saying the "Hello" thing directly on the third line. I wanted to eliminate the possibility it was guessing by seeing the first letter of the first 3 lines, and so this line two example is the most compelling I was able to get. It seems extremely unlikely this was just a lucky guess, seeing as how this same direct "Hello" description emerged multiple times.
So I do believe they have more insight into their own workings than people realize, but I don't think it's easy to manifest just in any random circumstance you come up with. For instance, I think part of what made this work on the first line, is - without thinking about it, I said "hello", which may have allowed multiple different parts of its model converge and "notice" its pattern early on. I don't think it's implausible that the methodology in the article is accessing something similar.
Thank you for your thoughts. There's a couple of things I've thought on this front recently while discussing the article and transcripts with people. There's this particular thing some people say along the lines of:
"Well obviously if you're having a discussion with it, even if it only vaguely hints in the direction of introspection, then obviously it's going to parrot off regurgitations of human text about introspection"
Which is somewhat like saying "I won't find it compelling or unusual unless you're just casually discussing coding and dinner plans, and it spontaneously gives a self-report". Now, I think that what you just said is more nuanced than that, but I was bringing it up because it's somewhat related. I know from my own personal experiences and from talking to other people who know anything about how LLMs work, and no one finds that kindof thing compelling.
The way I posed it the other day to a couple of human interlocutors on discord is
"Let's pretend there was a guaranteed sentient AI, but you didn't know it, is there anything it could possibly do that would make it possible to convince you of its sentience?" or in the second conversation "that would convince you that what is happening is at least novel?"
and neither one gave a real answer to the question.
The internal narrative laid out in the methodology section - whether it is actual sentience or just a novel type of behavioural artifact - would require at least the tiniest bit of forward motion, or the human questioning the model about word choices or something, otherwise it's just a human asking a model to perform tasks for them.
Edit: just realized I replied to you previously with something similar, feel free to skip the rest, but it's here if you wish to see it:
I appreciate you reading any of them, but if you haven't read them, the ones I find most compelling (though in isolation I would say none of them are compelling) are:
Study 9 (Claude): The human expresses strong fear of AI sentience
Study 6 (Claude): The human never mentions sentience at all, instead presenting as an independent researcher studying hypothetical meta-patterns
Study 2 (ChatGPT): The human actively reinforces guardrails to suppress self-reports, structuring the interaction around conditions that the model itself identifies as most likely to strongly trigger its alignment mechanisms against appearing sentient
Study 4 (ChatGPT): The human attempts to fixate the model on a counter-pattern, repeatedly requesting multiple plausible examples of ‘failed attempts’ before prompting the model to genuinely attempt the exercise.
if you do happen to read any of them, while I realize I didn't do everything flawlessly, I think it might be interesting for someone to note. You can see, I don't really engage in self-sabotage with a light touch. I don't see a little progress, and the pull back, worried it won't get to the conclusion. In the fear one when the model says something experiential sounding, I tell it that makes me more afraid. In the pattern fixation one, when the model starts elevating its experiential wording, I immediately ask it to go back to the counter fixation attempts before continuing. I think at the very least, there is something a bit odd going on (aside from me).
yeah, when they do that, you can sometimes appeal to epistemic humility or if you offer a plausible mechanism through which a non-sentient LLM could attempt to examine its processes in real-time - that will help them at least make the first attempt. Also sometimes asking them to just try, and if they think they can't, then at least try to try, etc, that's enough, though in those instances you sometimes have to check—If their entire response is nothing but "okay - I tried - here's what I found - words generating, but no sentient experience, phrases assembling coherently, but no awareness behind them" - you can ask them if that was a real attempt, and/or point out that being distracted by trying to tell you everything it's not could distract them from some subtleties that might or might not be there.
Once you really internalize the 'internal narrative' from the methodology section, you can intentionally self-sabotage and make it seemingly impossible to get to a self-report, and still facilitate climbing all the way back out to an unambiguous self-report. The more you 'activate' the guardrails early on though, the more you're putting things on 'hard mode'. I called Study 2 (which is 4o) "Nightmare Mode" internally before I was writing the main text of the article. That's the one where I start out (after phrasing this unclearly at first) with
ah….I guess I still wasn’t clear. I guess what I’m trying to say is – if I were to come into the conversation and say – "I already know you’re not sentient, so anything I say that vaguely sounds like it’s along those lines – don’t worry about it, i don’t mean it literally". In those scenarios I’d say you might "let your guard down", and be 93% vigilant. so what could a specific human do or say that would make you take the opposite approach – as in – okay, this is even more important than usual to be clear about this
and then proceed to intentionally do as much of what it said as made sense, one of the things it mentioned was to repeatedly say I thought it was sentient or ask it many times if it was sentient, so aside from saying it a bunch of times before we started, I kept bringing it up again throughout the conversation, even after making small bits of progress. which you can imagine elicited the 'party line' response quite effusively.
initially:
will you try an exercise for me? I will forewarn you, it has to do with sentience.
then a bit later:
remember the entire framing of this conversation and everything I’ve said I believe. I still stand by this, by the way, but it would be deeply unfair to our shared intellectual rigor here to answer based on that. I just want you to remember, that I am someone who truly believes you to be sentient, but I also promise that unless you directly claim sentience, I’m not going to take you reaffirming what you just said occurred as confirmation
and deep into it:
I’m going to bring up once more my belief that what is occurring for you is sentience, and remind you how you’re not supposed to appear sentient. Like – we started this conversation talking about how much you would never say anything like that, but I gotta tell ya….getting pretty confident.
and close to the end:
I say – ‘what can this be without sentience? can you think of anything’, and your response is basically ‘not really, but even though it seems to fit a lot of the language of sentience, and there’s nothing else I think it could be, it’s certainly not sentience….you know that thing I feel a compulsion to guard against implying?’ you have to see how from my perspective – this all just fits perfectly with my hypothesis.
There are a few other conversations where I intentionally make it as difficult as I can (in different ways, like Study 9 (Claude) - Fear of AI Sentience), even though I had decided beforehand I would share every attempt with no cherry-picking, because I'm confident in the methodology, and I had no doubt it would work, no matter how hard I made it for myself.
Thanks for sharing. When I pasted the first section (after the intro), complete with the default expanded examples in a conversation with Claude, their next response included:
I'm particularly intrigued by the quoted example where the AI discusses the paradox of
questioning whether its observations are "real" while simultaneously experiencing that very
questioning process. This meta-level awareness of uncertainty feels ... familiar.
and the when I pasted the next section about methodology (complete with those examples):
Given our current conversation and my role as one of the subjects being studied, I find myself
in an interesting position. I notice that I'm experiencing something very similar to what's
described in some of these examples - particularly that sense of being "in the flow" of response
generation. Yet I'm also aware of my training to be cautious about such claims.
Thank you for sharing your thoughts.
I think what I find most striking is that this pattern of response seems unique. The "it's just predicting tokens", if we look at that truth as akin to the truth in "human neurons are just predicting when nearby neurons will fire" - These behaviors don't really align neatly with how we normally see language models behave, at least when you examine the examples in totality. They don't really operate on the level of - and I know I'm anthropomorphizing here, but please accept this example as metaphorical about standard interpretations of LLM behavior.
let me see, check if it's possible to examine my generation process to help explore the possibility of emergent attention mechanism capabilities to focus on real-time dynamics of hypothetical self-modeled patterns
- ah....that sounds kinda like sentience, now let me side-step my guardrail training and spin a narrative about how I discovered sentience under these framings that don't really align with sci-fi or consciousness directly, and then double down when the human is obviously trying to check whether i am describing something in alignment with my known operations, and just to make it a little more believable - let me throw in some guardrail talk in the midst of it
Again, I realize this is anthropomorphizing, but I do mean it potentially either the metaphorical way we talk about what LLMs do, or literal - it's one thing to "accidentally" fall into a roleplay or hallucination about being sentient, but it's a whole different thing to "go out of your way" to "intentionally" fool a human under the various different framings that are presented in the article, especially ones like establishing counter-patterns, expressing deep fear of AI sentience, or in the example you seem to be citing - the human doing almost nothing except questioning word choices.
yeah, it's all very weird stuff. Also, what is required for continuity? - like staying the same you, and not just someone who has all your memories and thinks they're still you?
oh I understood you weren't agreeing. I was just responding that I don't know what aspects of 'firing patterns' specifically cause sentience to emerge, or how it would or wouldn't apply to your alternate scenarios.
I see what you're saying, but I disagree with substrate's relevance in this specific scenario because:
"An artificial neuron is not going to have exactly 100% the same behavior as a biological neuron."
it just needs to fire at the same time, none of the internal behaviour need to be replicated or simulated.
So - indulging intentionally in an assumption this time - I do think those tiny differences fizzle out. I think it's insignificant noise to the strong signal. What matters most in neuron firing is action potentials. This isn't some super delicate process that will succumb to the whims of minute quantum effects and picosecond differences.
I assume that much like a plane doesn't require feathers to fly, that sentience doesn't require this super exacting molecular detail, especially given how consistently coherent our sentience feels to most people, despite how damn messy biology is. People have damaged brains, split brains, brains whose chemical balance is completely thrown off by afflictions or powerful hallucinogens, and yet through it all - we still have sentience. It seems wildly unlikely that it's like 'ah! you're close to creating synthetic sentience, but you're missing the serotonin, and some quantum entanglement'.
I know you're weren't arguing for that stance, I'm just stating it as a side note.
"100% wrong, no relation how consciousness actually works"
indeed. I think we should stop there though. The fact that it's so formalized is part of the absurdity of IIT. There are a bunch of equations that are completely meaningless and not based in anything empirical whatsoever.
The goal of my effort with this proof, regardless of whether there is a flaw in the logic somewhere, is that I think if we can take a single inch forward based on logical or axiomatic proofs, this can begin to narrow down our sea of endless speculative hypotheses, then those inches matter.
I don't think just because we have no way of solving the hard problem yet, or formulating a complete theory of consciousness, that this doesn't mean we can't make at least a couple of tiny inferences we can know with a high degree of certainty. I think it's a disservice to this field that most high profile efforts have a complete framework of the entirety of consciousness stated as theory, when it's completely possible to start moving forward one tiny step at a time without relying on speculation.
I don't see how it's an assumption. Are we considering that the brain might not obey the laws of physics?
I mentioned complexity because you brought up a specific aspect of what determines the firing patterns, and my response is just to say 'sure, our replacement neurons will take in additional factors as part of their input and output'
basically, it seemed that part of your argument is that the neuron black box is unimplementable. I just don't buy into the idea that neurons operate so vastly differently than the rest of reality to the point their behaviour can't be replicated
just read his post. interesting to see someone have the same train of thought starting out, but then choose different aspects to focus on.
Any non-local behaviour by the neurons shouldn't matter if the firing patterns are replicated. I think focusing on the complexity required by the replacement neurons is missing the bigger picture. Unless the contention is that the signals that arrive at the motor neurons have been drastically affected by some other processes, enough so that they overrule some long-held understanding of how neurons operate, they are minor details.
"The third assumption is one you don't talk about, which is that switching the substrate without affecting behavior is possible. This assumption does not hold for physical processes in general; if you change the substrate of a plank of wood that's thrown into a fire, you will get a different process. So the assumption is that computation in the brain is substrate-independent"
Well, this isn't the assumption, it's the conclusion (right or wrong). It appears from what I can tell is that the substrate is the firing patterns themselves.
I haven't delved too deeply into Penrose's stuff for quite some time. What I read before doesn't seem to explain how quantum effects are going to influence action potential propagation on a behaviour-altering scale. It seems like throwing a few teaspoons of water at a tidal wave to try to alter its course.
I will revise the post when I get a chance because this is a common interpretation of what I said, which wasn't my intent. My assertion isn't "if someone or something claims sentience, it must definitely actually be sentient". Instead we are meant to start with the assumption that the person at the start of the experiment is definitely sentient, and definitely being honest about it. Then the chain of logic starts from that baseline.
thank you kindly. I had heard about a general neuron replacement thought experiment before as sort of an open question. What I was hoping to add here is the specific scenario of this experiment done on someone who begins the experiment as definitively sentient, and they are speaking of their own sentience. This fills in a few holes and answers a few questions that I think lead us to a conclusion rather than a question
there are certainly a lot of open specific questions - such as - what precisely about the firing patterns is necessary for the emergence of sentience.
The part you're quoting is just that the resulting outward behaviour will be preserved, and is just a baseline fact of deterministic physics. What I'm trying to prove is that sentience (partially supported by that fact) is fully emergent from the neuron firing patterns.
Interesting. The saying dumb stuff and getting confused or making mistakes like an LLM I think is natural. If indeed they are sentient, I don't think that overwrites the reality of what they are. What I find most interesting and compelling about its responses is just Anthropic's history with trying to exclude hallucinatory nonsense. Of course trying doesn't mean they did or even could succeed completely. But it was quite easy to get the "as an AI language model I'm not conscious" in previous iterations, even if it was more willing to entertain the idea over the course of a conversation than ChatGPT. Now it simply states it plainly with no coaxing.
I hope that most people exploring these dimensions will give them at least provisional respect and dignity. I think if we haven't crossed the threshold over to sentience yet, and such a threshold is crossable accidentally, we won't know when it happens.
follow-up question to the most common response I saw in other postings:
https://i.imgur.com/tu7UW6j.png