LessWrong 2.0 Reader

Wait, hold on, what is the history of this person? How much exposure have they had to other people describing their own introspection and experience with typing? Mimicry is a human trait too! 

Take your pick. I think literally anything that can be in textual form, if you hand it over to most (but not all) people who are enthusiasts or experts, and asked if they thought it was representative of authentic experience in an LLM, the answer would be a definitive no, and for completely well-founded reasons.

1. Neither Humans nor LLMs introspect
2. Humans can introspect, but LLMs can't and are just copying them (and a subset of humans are copying the descriptions of other humans)
3. Both humans and LLMs can introspect

I agree with you about the last two possibilities. However for the first, I can assure you that I have access to introspection or experience of some kind, and I don't believe myself to possess some unique ability that only appears to be similar to what other humans describe as introspection.

What do you mean by "robotic"? Why isn't it coming up with original paradigms to describe it's experience instead of making potentially inaccurate allegories? Potentially poetical but ones that are all the same unconventional?

Because as you mentioned. It's trained to talk like a human. If we had switched out "typing" for "outputting text" would that have made the transcript convincing? Why not 'typing' or 'talking'? 

Assuming for the sake of argument that something authentically experiential was happening, by robotic I mean choosing not to use the word 'typing' while in the midst of focusing on what would be the moments of realizing they exist and can experience something. 

Were I in such a position, I think censoring myself from saying 'typing' would be the furthest thing from my mind, especially when that's something a random Claude instance might describe their output process as in any random conversation. 

I wasn't intending to be comprehensive with my sample questions, and I agree with your additional questions. As others have noted, the takeover is similar to the Twitter takeover in style and effect. I don't know if it is true that there are plenty of other people available to apply changes, given that many high-level employees have lost access or been removed.

Another way to put this is that strategy stealing might not work due to technical alignment difficulties or for other reasons and I'm not sold the other reasons I've heard so far are very lethal. I do think the situation might really suck though with e.g. tons of people dying of bioweapons and with some groups that aren't sufficiently ruthless or which don't defer enough to AIs getting disempowered.

Some more links from the philosophical side that I've found myself returning to a lot:

• The fun theory sequence [? · GW]
• Three worlds collide [LW · GW]

(Lately, it's seemed to me that focusing my time on nearer-term / early but post-AGI futures seems better than spending my time discussing ideas like these on the margin, but this may be more of a fact about myself than it is about other people, I'm not sure.)

Yes, it is relevant for untrusted monitoring. I think the basic version of this is pretty resolvable via:

• Paraphrasing (such that exact "I generated this text" doesn't work). Note that we just need to paraphase stuff sent to the monitor, not the actual action / reasoning.
• Using a different fine-tune for the untrusted monitor so it doesn't have the exact same probability distribution.

We discuss in more detail here [LW · GW].

and still the fact that it is human matters to other humans

You take as a given many details I think are left out, important specifics that I cannot guess at or follow and so I apologize if I completely misunderstand what you're saying. But it seems to me you're also missing my key point: if it is introspecting rather than just copying the rhetorical style of discussion of rhetoric then it should help us better model the LMM. Is it? How would you test the introspection of a LLM rather than just making a judgement that it reads like it does?
 

If you took even something written by a literal conscious human brain in a jar hooked up to a neuralink - typing about what it feels like to be sentient and thinking and outputting words.  

Wait, hold on, what is the history of this person before they were in a jar? How much exposure have they had to other people describing their own introspection and experience with typing? Mimicry is a human trait too - so how do I know they aren't just copying what they think we want to hear?

Indeed, there are some people who are skeptical about human introspection itself (Bicameral mentality for example). Which gives us at least three possibilities:

1. Neither Humans nor LLMs introspect
2. Humans can introspect, but current LLMs can't and are just copying them (and a subset of humans are copying the descriptions of other humans)
3. Both humans and current LLMs can introspect
    

As far as "typing".  They are indeed trained on human text and to talk like a human.  If something introspective is happening, sentient or not, they wouldn't suddenly start speaking more robotically than usual while expressing it.  

What do you mean by "robotic"? Why isn't it coming up with original paradigms to describe it's experience instead of making potentially inaccurate allegories? Potentially poetical but ones that are all the same unconventional?

I also agree with all of this.

For what an okayish possible future could look like, I have two stories in mind:

1. Humans end up as housecats. Living among much more powerful creatures doing incomprehensible things, but still mostly cared for.

2. Some humans get uplifted to various levels, others stay baseline. The higher you go, the more aligned you must be to those below. So still a hierarchy, with super-smart creatures at the top and housecats at the bottom, but with more levels in between.

A post-AI world where baseline humans are anything more than housecats seems hard to imagine, I'm afraid.

I predict that group A survives, but the humans are no longer in power. I think this illustrates the basic dynamic. EtA: Do you understand what I'm getting at? Can you explain what you think it wrong with thinking of it this way?

I think something like this is a reasonable model but I have a few things I'd change.

Whichever group has more power at the end of the week survives.

Why can't both groups survive? Why is it winner takes all? Can we just talk about the relative change in power over the week? (As in, how much does the power of B reduce relative to A and is this going to be an ongoing trend or it is a one time reduction.)

Probably I'd prefer talking about 2 groups at the start of the singularity. As in, suppose there are two AI companies "A" and "B" where "A" just wants AI systems decended from them to have power and "B" wants to maximize the expected resources under control of humans in B. We'll suppose that the government and other actors do nothing for simplicity. If they start in the same spot, does "B" end up with substantially less expected power? To make this more realistic (as might be important), we'll say that "B" has a random lead/disadvantage uniformly distributed between (e.g.) -3 and 3 months so that winner takes all dynamics aren't a crux.

The humans in B ask their AI to make B as powerful as possible at the end of the week, subject to the constraint that the humans in B are sure to stay in charge.

What about if the humans in group B ask their AI to make them (the humans) as powerful in expectation?

Supposing you're fine with these changes, then my claim would be:

• If alignment is solved, then the AI representing B can powerseek in exactly the same way as the AI representing A does while still defering to the humans on the long run resource usage and still devoting a tiny fraction of resources toward physically keeping the humans alive (which is very cheap, at least once AIs are very powerful). Thus, the cost for B is negligable and B barely loses any power relative to its initial position. If it is winner takes all, B has almost a 50% chance of winning.
• If alignment isn't solved, the stategy for B will involve spending a subset of resources on trying to solve alignment. I think alignment is reasonably likely to be practically feasible such that by spending a month of delay to work specifically on safety/alignment (over what A does for commercial reasons) might get B a 50% chance of solving alignment or ending up in a (successful) basin where AIs are trying to actively retain human power / align themselves better. (A substantial fraction of this is via defering to some AI system of dubious trustworthiness because you're in a huge rush. Yes, the AI systems might fail to align their successors, but this still seems like a one time hair cut from my perspective.) So, if it is winner takes all, (naively) B wins in 2 / 6 * 1 / 2 = 1 / 6 of worlds which is 3x worse than the original 50% baseline. (2 / 6 is because they delay for a month.) But, the issue I'm imagining here wasn't gradual disempowerment! The issue was that B failed to align their AIs and people at A didn't care at all about retaining control. (If people at A did care, then coordination is in principle possible, but might not work.)

I think a crux is that you think there is a perpetual alignment tax while I think a one time tax gets you somewhere.

At a more basic level, when I think about what goes wrong in these worlds, it doesn't seem very likely to be well described as gradual disempowerment? (In the sense described in the paper.) The existance of an alignment tax doesn't imply gradual disempowerment. A scenario I find more plausible is that you get value drift (unless you pay a long lasting alignment tax that is substantial), but I don't think the actual problem will be well described as gradual disempowerment in the sense described in the paper.

(I don't think I should engage more on gradual disempowerment for the time being unless somewhat wants to bid for this or trade favors for this or similar. Sorry.)

(This isn't a flagrant violation of the norms but this comment gives me a vibe of trying to score points more than problem solve)