LessWrong 2.0 Reader

I find myself strongly disagreeing with what is being said in your post. Let me preface by saying that I'm mostly agnostic with respect to the possible "explanations" of consciousness etc,  but I think I fall squarely within camp 2. I say mostly because I  lean moderately towards physicalism.

First, an attempt to describe my model of your ontology:

You implicitly assume that consciousness / subjective experience can be reduced to a physical description of the brain, which presumably you model as a classical (as opposed to quantum) biological electronic circuit.  Physically, to specify some "brain-state" (which I assume is essentially the equivalent of a "software snapshot" in a classical computer) you just need to specify a circuit connectivity for the brain, along with the currents and voltages between the various parts of the circuit (between the neurons let's say). This would track with your mentions of reductionism and physicalism and the general "vibe" of your arguments.  In this case I assume you treat conscious experience roughly as "what it feels like" to be software that is self-referential on top of taking in external stimuli from sensors. This software is instantiated on a biological classical computer instead of a silicon-based one.

With this in mind, we can revisit the teleporter scenario. Actually, let's consider a copier instead of a teleporter, in the sense that you dont destroy the original after finishing the procedure. Then, once a copy is made, you have two physical brains that have the same connectivity, the same currents and the same voltages between all appropriate positions. Therefore, based on the above ontology, the brains are physically the same in all the ways that matter and thus the software / the experience is also the same. (Since software is just an abstract "grouping" which we use to  refer to the  current physical state of the hardware)

Assuming this captures your view, let me move on to my disagreements:

My first issue with your post is that this initial ontological assumption is neither mentioned explicitly nor motivated. Nothing in your post can be used as proof of this initial assumption. On the contrary,  the teleporter argument, for example,  becomes simply a tautology if you start from your premise - it cannot be used to convince someone that doesn't already subscribe to your views on the topic. Even worse, it seems to me that your initial assumption forces you to contort (potential) empirical observation to your ontology, instead of doing the opposite.

To illustrate, let's assume we have the copier - say it's a room you walk into, you get scanned and then a copy is reconstructed in some other room far away.  Since you make no mention of quantum, I guess this can be a classical copy, in the sense that it can copy essentially all of the high-level structure, but it cannot literally copy the positions of specific electrons, as this is physically impossible anyways. Nevertheless, this copier can be considered "powerful" enough to copy the connectivity of the brain and the associated currents and voltages. Now, what would be the experience of getting copied, seen from a first-person, "internal", perspective? I am pretty sure it would be something like: you walk into the room, you sit there, you  hear say the scanner working for some time, it stops, you walk out. From my agnostic perspective, if I were the one to be scanned it seems like nothing special would have happened to me in this procedure. I didnt feel anything weird, I didnt feel my "consciousness split into two" or something. Namely, if I consider this procedure as an empirical experiment, from my first person perspective I dont get any new / unexpected observation compared to say just sitting in an ordinary room. Even if I were to go and find my copy, my experience would again be like meeting a different person which just happens to look like me and which claims to have similar memories  up to the point when I entered the copying room. There would be no way to verify or to view things from their first person perspective.

At this point, we can declare by fiat that me and my copy are the same person / have the same consciousness because our brains, seen as classical computers, have the same structure, but this experiment will not have provided any more evidence to me that this should be true. On the contrary, I would be wary to, say, kill myself or to be destroyed after the copying procedure, since no change will have occured to my first person perspective, and it would thus seem less likely that my "experience" would somehow survive because of my copy.

Now you can insist that philosophically it is preferable to assume that brains are classical computers etc, in order to retain physicalism which is preferable to souls and cartesian dualism and other such things. Personally, I prefer to remain undecided, especially since making the assumption brain= classical hardware, consciousness=experience as software leads to weird results. It would force me to conclude that the copy is me even though I cannot access their first person perspective (which defeats the purpose) and it would also force me to accept that even a copy where the "circuit" is made of water pipes and pumps, or gears and levers also have an actual, first person experience as "me", as long as the appropriate computations are being carried out.  

One curious case where physicalism could be saved and all these weird conclusions could be avoided would be if somehow there is some part of the brain which does something quantum, and this quantum part is the essential ingredient for having a first person experience. The essence would be that, because of the no-cloning theorem, a quantum-based consciousness would be physically impossible to copy, even in theory. This would get around all the problems which come with the copyability implicit in classical structures. The brain would then be a hybrid of classical and quantum parts, with the classical parts doing most of the work (since neural networks which can already replicate a large part of  human abilities are classical) with some quantum computation mixed in, presumably offering some yet unspecified fitness advantage. Still, the consensus is that it is improbable that quantum computation is taking place in the brain, since quantum states are extremely "fragile" and would decohere extremely rapidly in the environment of the brain...

Wow - Didn't expect to see that headline (other than 2 weeks ago)

it’s very possible that it could become a practical problem at some point in the future.

I kind of doubt it.  Practical problems will have complexity and details that overwhelm this simple model, making it near-irrelevant.  Alternately, it may be worth trying to frame a practical decision that an individual or small group (so as not to have to abstract away crowd and public choice issues) could make where this is important.

Do you think a logarithmic scale makes more sense than a linear scale?

Yes, but it probably doesn't fix the underlying problem that quantifications are unstable and highly variable across agents.

Wouldn't it follow that in the same way you anticipate the future experiences of the brain that you "find yourself in" (i.e. the person reading this) you should anticipate all experiences, i.e. that all brain states occur with the same kind of me-ness/vivid immediacy?

What's the empirical or physical content of this belief?

I worry that this may be another case of the Cartesian Ghost rearing its ugly head. We notice that there's no physical thingie that makes the Ghost more connected to one experience or the other; so rather than exorcising the Ghost entirely, we imagine that the Ghost is connected to every experience simultaneously.

But in fact there is no Ghost. There's just a bunch of experience-moments implemented in brain-moments.

Some of those brain-moments resemble other brain-moments, either by coincidence or because of some (direct or indirect) causal link between the brain-moments. When we talk about Brain-1 "anticipating" or "becoming" a future brain-state Brain-2, we normally mean things like:

• There's a lawful physical connection between Brain-1 and Brain-2, such that the choices and experiences of Brain-1 influence the state of Brain-2 in a bunch of specific ways.
• Brain-2 retains ~all of the memories, personality traits, goals, etc. of Brain-1.
• If Brain-2 is a direct successor to Brain-1, then typically Brain-2 can remember a bunch of things about the experience Brain-1 was undergoing.

These are all fuzzy, high-level properties, which admit of edge cases. But I'm not seeing what's gained by therefore concluding "I should anticipate every experience, even ones that have no causal connection to mine and no shared memories and no shared personality traits". Tables are a fuzzy and high-level concept, but that doesn't mean that every object in existence is a table. It doesn't even mean that every object is slightly table-ish. A photon isn't "slightly table-ish", it's just plain not a table.

Which just means, all brain states exist in the same vivid, for-me way, since there is nothing further to distinguish between them that makes them this vivid, i.e. they all exist HERE-NOW.

But they don't have the anticipation-related properties I listed above; so what hypotheses are we distinguishing by updating from "these experiences aren't mine" to "these experiences are mine"?

Maybe the update that's happening is something like: "Previously it felt to me like other people's experiences weren't fully real. I was unduly selfish and self-centered, because my experiences seemed to me like they were the center of the universe; I abstractly and theoretically knew that other people have their own point of view, but that fact didn't really hit home for me. Then something happened, and I had a sudden realization that no, it's all real."

If so, then that seems totally fine to me. But I worry that the view in question might instead be something tacitly Cartesian, insofar as it's trying to say "all experiences are for me" -- something that doesn't make a lot of sense to say if there are two brain states on opposite sides of the universe with nothing in common and nothing connecting them, but that does make sense if there's a Ghost the experiences are all "for".

I think that paper is some evidence that there's typically no huge effect from internal activations being optimized for predicting future tokens (on natural language). But I don't think it's much (if any) evidence that this doesn't happen to some small extent or that it could be a huge effect on certain other natural language tasks.

(In fact, I think the myopia gap is probably the more relevant number than the local myopia bonus, in which case I'd argue the paper actually shows a pretty non-trivial effect, kind of contrary to how the authors interpret it. But I haven't read the paper super closely.)

Also, sounds like you're aware of this, but I'd want to highlight more that the paper does demonstrate internal activations being optimized for predicting future tokens on synthetic data where this is necessary. So, arguably, the main question is to what extent natural language data incentivizes this rather than being specifically about what transformers can/tend to do.

In that sense, thinking of transformer internals as "trying to" minimize the loss on an entire document might be exactly the right intuition empirically (and the question is mainly how different that is from being myopic on a given dataset). Given that the internal states are optimized for this, that would also make sense theoretically IMO.

“Imagine a square circle, and now answer the following questions about it…”.

Just use the Chebyshev (aka maximum or $L^{\infty }$) metric.

As a test, I asked a non-philosopher friend of mine what their view is. Here's a transcript of our short conversation: https://docs.google.com/document/d/1s1HOhrWrcYQ5S187vmpfzZcBfolYFIbeTYgqeebNIA0/edit 

I was a bit annoyingly repetitive with trying to confirm and re-confirm what their view is, but I think it's clear from the exchange that my interpretation is correct at least for this person.

I'm confused, is the death to discomfort comparison based on the cumulative experience that the loved ones and friends of a person who has died might experience in grief and despair that someone they cared about died? Or are you suggesting that a death is a superlatively uncomfortable event for the individual who is dying?

I can't see a way of making discomfort to death fungible, at least partly because to experience discomfort requires someone to continue on living. 

I expect a lot more open releases this year and am committed to test their capabilities and safety guardrails rigorously.

Glad you're planning on continual testing, that seems particularly important here, where the default is every once in awhile some new report comes out with a single data point about how good some model is and people slightly freak out. Having the context of testing numerous models over time seems crucial for actually understanding the situation and being able to predict upcoming trends. Hopefully you have and will continue to find ways to reduce the effort needed to run marginal experiments, e.g., having a few clearly defined tasks you repeatedly use, reusing finetuning datasets, etc. 

Is there even anybody claiming there is an experiential difference?

Yep! Ask someone with this view whether the current stream of consciousness continues from their pre-uploaded self to their post-uploaded self, like it continues when they pass through a doorway. The typical claim is some version of "this stream of consciousness will end, what comes next is only oblivion", not "oh sure, the stream of consciousness is going to continue in the same way it always does, but I prefer not to use the English word 'me' to refer to the later parts of that stream of consciousness".

This is why the disagreement here has policy implications: people with different views of personal identity have different beliefs about the desirability of mind uploading. They aren't just disagreeing about how to use words, and if they were, you'd be forced into the equally "uncharitable" perspective that someone here is very confused about how relevant word choice is to the desirability of uploading.

The alternative to this is that there is a disagreement about the appropriate semantic interpretation/analysis of the question. E.g. about what we mean when we say "I will (not) experience such and such". That seems more charitable than hypothesizing beliefs in "ghosts" or "magic".

I didn't say that the relevant people endorse a belief in ghosts or magic. (Some may do so, but many explicitly don't!)

It's a bit darkly funny that you've reached for a clearly false and super-uncharitable interpretation of what I said, in the same sentence you're chastising me for being uncharitable! But also, "charity" is a bad approach to trying to understand other people [LW · GW], and bad epistemology can get in the way of a lot of stuff.