LessWrong 2.0 Reader

I also wonder what Buck thinks about CoTs becoming less transparent (especially in light of recent o1 developments).

Parts of human mind are not little humans. They are allowed to be irrational. It can't be rational subagents all the way down. Rationality itself is probably implemented as subagents saying "let's observe the world and try to make a correct model" winning a reputational war against subagents proposing things like "let's just think happy thoughts".

But I can imagine how some subagents could have less trust towards "good intentions that didn't bring actual good outcomes" than others. For example, if you live in an environment where it is normal to make dramatic promises and then fail to act on them. I think I have read some books long ago claiming that children of alcoholic parents are often like that. They just stop listening to promises and excuses, because they have already heard too many of them, and they learned that nothing ever happens. I can imagine that they turn this habitual mistrust against themselves, too. That "I tried something, and it was a good idea, but due to bad luck it failed" resembles too much the parent saying how they had the good insight that they need to stop drinking, but only due to some external factor they had to drink yet another bottle today. Shortly, if your environment fails you a lot, as a response you can become unrealistically harsh on yourself.

Another possible explanation is that different people's attention is focused on different places. Some people pay more attention to the promises, some pay more attention to the material results, some pay more attention to their feelings. This itself can be a consequence of the previous experience with paying attention to different things.

Thanks for making a well-thought out comment. It's really helpful for me to have an outside perspective from another intelligent mind.

I'm hoping to learn more from you, so I'm going to descend into a way of writing that assumes we have a lot of the same beliefs/understandings about the world. So if it gets confusing, I apologize for not being able to communicate myself more clearly.



Your 1st point:
This is an interesting perspective shift. The concept that by endeavoring to help people understand suffering, I would be causing suffering itself, since I'd be creating an expectation/ideal that virtually no one in society has met. I agree with this point, and I have a lot of perspectives on this. I'm curious about what you think of them.

My 1st perspective:  The marketing perspective.

Since I want people to change, I want them to be in pain or suffer. Since without pain, the motivation to change is typically weak or non-existent. As cruel an non-idealistic as it may be, I don't endeavor to be an idealist. I endeavor to be realist and rationalist. People have to feel pain in order to take actions necessary to better their lives. No one would imply that a farmer shouldn't put in the work to plant crops. The village's desire to eat, is of greater importance than the farmer's desire to feel physically comfortable.

I think that if I were trying to cater to idealist perspectives, I would need to emphasize suffering being bearable when in pursuit of reward. On some level, I understand how this process works in my own psychology and have been able to manipulate it, but my ability to help other people do the same thing remains uncertain. I think that there are multiple other sources of reward that can be used for this purpose other than goal-progress oriented reward, (Such as food, media entertainment, or social) but I agree that showing people that they are making clear progress will be very beneficial for increasing the acceptability of a person's suffering in this context.

My 2nd perspective: Which subsets of people will experience increased suffering as a result of the implementation of the ideas expressed in this document?

I don't think this would negatively impact people who are currently already unsatisfied with their life. Since it would just shift their focus from their current ideals/unfulfilled desires to the new one that's being presented. 

(Except this time, since they understand the systems at play in their mind and in their environment, they will actually have the power to change things)

So people won't need to rely on a large set of trial & error + heuristics to perceive that they've reached their ideal. 

I could be wrong on the aforementioned point. If instead of shifting focuses, a person adds this to a long subconscious list of things they should be that they are not, it could increase suffering even in people who are already dissatisfied with life. If testing reveals that this is the case, I think that unraveling sets of beliefs about what a person should be will help alleviate this suffering. (This would be done by utilizing marketing tactics such as storytelling)

but for subsets of people who are currently satisfied with their life, I don't think that introducing new ideals will increase suffering. There are lots of current societal ideals/expectations that re near-impossible to live up to. If this subset of people are satisfied even in spite of the external ideals/expectations they are unable to embody, I see no reason to believe that introducing a new ideal would cause this subset of people to suffer.

Although It could be argued that the reason these people are satisfied is because they've fooled themselves into believing they are meeting those impossible societal ideals/expectations. Even if this is argued, I think they would just continue to fool themselves by using the mental processes that have reduced their suffering in the past.

My 3rd perspective: Worst case scenario:

I have some critically inaccurate belief about the world/other people. And in reality, the only thing I'll be able to do is show people how life could be better without actually being able to get anyone to change in this way.


Your 2nd point

Completely agree. It's good that you caught me on this point, because in reflection, if I don't clarify the core of the problem more clearly, the problem could easily be mispercieved as something far more simple than is necessary to actually improve people's lives. I almost fell into the"One man's utopia is another man's hell" archetype. I think I'm just unaware about how to implement this without getting too deep into psychology and a lot of interrelated concepts.

If I was solely speaking to an audience that had your level of understanding of human psychology as it relates to suffering, then I could immediately begin clarifying my position on deeper concepts related to suffering.

But since the average person doesn't have a concept of suffering as an isolated concept from pain, I think that it could be difficult to help people make that conceptual jump.

I do intend to make a few separate versions intended for different target audience levels of intellect, so I might be able to solve this problem with my implementation here.


Your 3rd point
In retrospect, I realize that I was very liberal with my usage of the term "EA", and I made no effort to clarify what I meant by my usage of the term. Just to be sure we're on the same page, When I say "We" in "we are products of EA", I'm referring to a hypothetical group of people who want to prevent human extinction. (The target audience of this post). I definitely don't mean to imply that another person or group's altruistic deliberations had anything to do with our current beliefs or abilities. I'm also not referring to any organization or group of people when I say "EA", What I mean is the broader state of altruistic tendencies among intellectuals/rational people as a movement.

I realize now that I made an unjustified assumption that efforts for the prevention of human extinction would have altruistic motivations. And as you've pointed out, I may be wrong.


"Would reducing suffering really be a good thing?"

 I'm thinking about this question. And I realize I'm not actually qualified to answer it.

From a humanitarian or idealist perspective, obviously we should reduce pointless suffering and help people gain the tools necessary to deal with or accept suffering. But from a complexity or future-prediction perspective, it is difficult to know what the far-reaching consequences of this hypothetical world would be. If people know how to consciously reduce their own suffering and engineer their environments in a way that makes life more satisfactory for them, the results could be akin to a sort of malignant idea virus. As we've both said, without suffering, there is no change, By democratizing these tools, we could be erasing a core and crucial element of human change. Since choosing suffering as opposed to satisfaction is completely contradictory to human nature, once a person receives the information, they could be forever changed for the worse.


I also grew because of my suffering. And In this post, I meant to say that I was the most productive when I didn't eat. Not eating only significantly reduced my productivity when how much I ate was outside of my conscious control.

Suffering has made me into the person I am, and I am an avid believer in the pro-social benefits of personal suffering. But it is still true that unacceptable suffering outside of an individual's control is what causes the scarcity mindset. (Eradicating the scarcity mindset & increasing the prevalence of altruistic perspectives is the whole point of reducing human suffering in the context of preventing human extinction.) 

Additionally, there are certain important aspects of human psychology that I'm still unsure about. For example, to what degree does pain as a stimulus cause change? To what degree does suffering cause change?  To what degree does an individual's acceptability of suffering affect change?

And to the 3 aforementioned questions, in what way?

Your 4th point

My fault for not better clarifying my perspective here.

I'm claiming that even if scarcity mindsets fade and altruism spreads, we will still go extinct if the average person is as dumb as I am. "Create a community", and "Spread logical decisionmaking" are complementary to the concept of reducing human suffering, but their ultimate purpose is the continued existence of the human species. They are 2 of 3 points aimed at managing humanity's likelihood of destroying itself with some form of technology.

I personally think that intellect past a certain level gives humans the ability to deliberately manipulate their suffering, but I think that on the spectrum with which every human I'm aware of exists lies, intellect does not seem to make suffering controllable in any non-proxy way. In other words, I don't think that tech or intellect are strong predictors relative to suffering at our current capabilities for tech and intellect.

To respond to your point on anything other than survival & reproduction being a fabricated problem, I would agree. I feel like you can always meta-contradict the concept of meaning, since human meaning is constructed by our psychological systems, rather than by any universal truth. We can argue that pain is bad, and pleasure is good. Only to realize that those two concepts are only proxies for what we actually optimize for, suffering and satisfaction. So then we can argue that suffering is bad, and satisfaction is good. Only to gain perspective on the concept of agency, and to realize that suffering and satisfaction are just mesa-optimizers for evolution. I believe this process can go on an on, but the reality is that we are still mesa-optimizers, and so we have a natural inclination towards certain things and an intuition to call it meaningful. Anyone can take a nihilistic perspective an argue that nothing is meaningful, but if nothing is meaningful, then there is also no point in doing nothing, and we can do whatever we want. So I think the concept of meaning is a sort of recursive argument that we don't have the intellectual tools to solve. So rather than do nothing, I think we should do what we can.
 
Your 5th point (Bureaucracy)
I don't know anything about how decentralization can help with the problem of bureaucracy. Maybe you can point me to a source of information that will help me see your perspective on this? 

I'm also interested in your perspective on a few entities within a bureaucracy hogging all the resources. I presume you're referring to management claiming credit or capital distributions received by the owner class?

I look at bureaucracy from a business perspective. They call it operational complexity, and it refers to the reduced level of control of the founder over a business as the organization/tasks get more complex.

As the founder loses control over incentive structures, hiring practices, and training, the impact of the founder's competence dilutes and regresses towards the mean of the average human. 

It also refers to increases in tasks that are not the actual revenue-producing activity. An example would be salesman filling out excel sheets instead of talking with prospects.

Another example would be the increased training time required for adding an additional task to a specific role, and the increased training time for the people who are training those people, and the butterfly effect that has on the entirety of the business.

Also, the increased need for competent problem-solving as old systems break, and new systems (Or slight variations of old systems) become necessary.

My shortest definition on how I see Bureaucracy = reduced efficiency in multiple ways for multiple reasons as organizations grow larger.

Your 6th point (Moloch)
From my understanding, Moloch = a more complex representation of the prisoner's dilemma. 

The state of everything growing worse as a result of competition & a lack of control resulting from the fact that you either pool resources in a locally helpful but globally harmful way, or you seize to exist.
 

Your take: "Caused by too much information and optimization, and therefore unlikely to be solved with information and optimization. My take here is the same as with intelligence and tech."

I wonder about what basis/set of information you're using to make these 3 claims? I am currently unable to respond in a productive way without further context.

"Why hasn't moloch killed us sooner? I believe it's because the conditions for moloch weren't yet reached (optimal strategies weren't visible, as the world wasn't legible and transparent enough), in which case, going back might be better than going forwards."

And I may have misunderstood your point here, but from my understanding you're arguing something like: "Why aren't we in a state of perfect Moloch in present society?" (I'm not sure what you mean by "optimal strategies weren't visible".)  And when you say going back instead of forwards, you seem to be implying that the solution to preventing human extinction as it relates to tech is simply to remove technology.

Which in my opinion, will inevitably lead future generations back to this exact point in technological capabilities, and they might not also decide to regress technological capabilities since their society would be vastly different than our own in terms of culture-based differences such as values & identity. I don't see the reasoning behind pushing the problem forward onto later generations as opposed to attempting to solve the problem for good.

It'd also be great if you could point me to a piece of writing on world legibility and transparency. Since I don't currently have the context with which to understand what those two things mean.

Your final point

What's leading humanity to extinction in my opinion: 
1. Increased technological capabilities
2. Our lack of ability to control the impacts of these capabilities

3. our lack of incentive to manage those capabilities.

4. Our lack of control over the first 3 points.

I agree that my proposed solutions for the problems we face are reliant on systems currently driving humanity towards extinction.

But I don't agree with the implied understanding that every system associated with our current path to extinction needs to be removed in order for human extinction to be prevented. I believe that certain aspects within the system can be changed, while larger high-level systems can remain the same. And we can still solve the problem of human extinction this way. For example, Elon Musk trying to get humans to mars is reliant on technology, but it is also a hedge against human-extinction at the same time.


I know I wrote a lot, but I love that you're making me question assumptions I've made that I've never even thought to question.

Maybe I don't currently have the base-knowledge necessary to create helpful high-level plans. But I'm not sure how I can better use my time relative to helping prevent human extinction.

I'll edit this document with what we've talked about here in mind and see what I can do to improve this post.

It seems like that assumption runs throughout the post though, eg 'But it won’t know what it actually tried, unless it is able to guess very accurately what it would have said', 'the AI might not even be able to get direct evidence of whether its attempts worked, or what its attempts even were.'

Those sentences are talking in the context of defer to resample, where I think it's actually reasonable to think the model won't see the resampled actions.

I don't just mean showing the model its past CoT. Current models are often given lots of information about their past actions and interactions so that they can produce better output; why would you expect that to change?

I'm specifically talking about not showing it actions that were resampled. You're totally right that they'll need other information about their actions and interactions (though it might be a good safety-usefulness tradeoff to somewhat redact this information).

Is the assumption that labs will be willing to accept worse performance because they recognize the model can't be trusted?

Yes, I am assuming that the deployer is willing to spend some performance in order to reduce risk from the model scheming.

See here for an updated version of my thinking based on discussion with Daniel:

https://www.lesswrong.com/editPost?postId=Tzdwetw55JNqFTkzK

Thanks for the great writeup.

Superposition ("local codes") require sparsity, i.e. that only few features are active at a time.

Typo: I think you meant to write distributed, not local, codes. A local code is the opposite of superposition.

Fair point. (I am not convinced by the argument that if the AI's are trained on human texts and feedback, they are likely to end up with values similar to humans, but that would be a long debate.)

Fischer's not against using it for tradeoffs, he's against using it as a singular indicator of worth. 

Saying that we should donate there as opposed to AMF, for example, I would argue is trolleying. You're making tradeoffs and implicitly saying this is worth as much as that. Perhaps you're giving lower trade offs than the pain pleasure stuff, but you didn't really mention these, and they seem important to the end claim "and for these reasons, you should donate to shrimp welfare." 

My understanding of the position that scheming will be unlikely is the following:

• Current LLMs don't have scary internalized goals that they pursue independent of the context they're in.
• Such beyond-episode goals also won't be developed when we apply a lot more optimization pressure to the models, given that we keep using the training techniques we're using today, since the inductive biases will remain similar and current inductive biases don't seem to incentivize general goal-directed cognition. Naturally developing deception seems very non-trivial [LW(p) · GW(p)], especially given that models are unlikely to develop long-term goals in pre-training [LW · GW].
• Based on the evidence we have, we should expect that the current techniques + some kind of scaffolding will be a simpler path to AGI than e.g. extensive outcome-based RL training. We'll get nice instuction-following tool AIs. [LW(p) · GW(p)] The models might still become agentic in this scenario, but since the agency comes from subroutine calls to the LLM rather than from the LLM itself, the classical arguments for scheming don't apply.
• Even if we get to AGI through some other path, the theoretical arguments in favor of deceptive alignment are flimsy [LW · GW], so we should have a low prior on other kinds of models exhibiting scheming.

I'm not sure about the other skeptics, but at least Alex Turner appears to believe [LW(p) · GW(p)] that the kind of consequentialist cognition necessary for scheming is much more likely to arise if the models are aggressively trained on outcome-based rewards, so this seems to be the most important of the cruxes you listed. This crux is also one of the two points on which I disagree most strongly with the optimists:

1. I expect models to be trained in outcome-based ways. This will incentivize consequentialist cognition and therefore increase the likelihood of scheming. This post makes a good case for this.
2. Even if models aren't trained with outcome-based RL, I wouldn't be confident that it's impossible for coherent consequentialist cognition to arise otherwise, so assigning deceptive alignment a <1% probability would still seem far-fetched to me.

However, I can see reasons why well-informed people would hold views different from mine on both of those counts (and I've written a long post [LW · GW] trying to explore those reasons), so the position isn't completely alien to me.