LessWrong 2.0 Reader

As a layman, I have not seen much unrealistic hype. I think the hype-level is just about right.

You should not bury such a good post in a shortform

Maybe it should be a game that everyone can play

You didn't ask me to pitch you but I will say a short pitch here for any bystanders. I know how to how find a handful of good people and I know how to let a good chef cook without isolating them either. And I can make a pretty good fried egg if we're starving.

As always, I may not be the intended audience, so please excuse my questions that might be patently obvious to the intended audience.

Am I right in understanding a very simplified version of this model is that if you use willpower too much without deriving any net benefits, eventually you'll suffer 'burnout' which really is just a mistrust of using willpower ever, which may have negative effects on other aspects of your life even where willpower is needed like, say, cleaning your house?

Willpower, as I understand it is another word for 'patience' or 'discipline', variously described as the ability to choose to endure pain (physical or emotional). Whether willpower actually exists is a question I won't get into here, let's assume for the sake of this model it does, and fits the description of the ability to choose to endure pain.

For me this sentence I find especially alien to me:

your psyche’s conscious verbal planner “earns” willpower (earns trust with the rest of your psyche) by choosing actions that nourish your fundamental, bottom-up processes in the long run. 

what is the "psyche's conscious verbal planner"?  I don't know what this is or what part of my mind, person, identity, totality as a organism or anything really that I can equate this label to. Also without examples of what actions are that nourish (again, would cleaning the house, cooking healthy meals be examples?), that are fundamental and those that aren't, it's even harder to pin down what this is and why you attribute willpower to it.

It appears to have the ability to force one's-self to go on a date, which really makes the "verbal" descriptor confusing since a lot of the processes that are involved in going on a date don't feel like they are verbal, lexical, or take the form of the speaker's native language written or spoken. At least in my experience, a lot of the thoughts, feelings, and motivations behind going on a date are not innately verbal for me and if you asked me "why did you agree to see this person?" - even if I felt no fear of embarrassment explaining my reasons - I'd have a hard time putting that into words. Or the words I'd use would be so impossibly vague ("they seem cool") as to suggest that there was a nonverbal reasoning or motivation.

Would this 'conscious verbal planner' also be the part of my mind and body that searches an online store a week later to see if those shoes I want are on special? Or would you attribute that to a different entity?

Is there an unconscious verbal planner?

When I am thinking very carefully about what I'm saying, but not so minutely that I'm thinking about the correct grammatical use, would the grammar I use be my unconscious verbal planner, while the content of my speech be the conscious verbal planner?

A lot of example, for me, of willpower often are nonverbal and come from guilt. Guilt felt as a somatic or bodily thing. I can't verbalize why I feel guilty, although it verbally equates to the words "should"  "must" and even "ought" when used as imperatives, not as modals. 
 

Ty for the link but these seem like both clearly bad semantics (e.g. under either of these the second-best hypothesis under consideration might score arbitrarily badly).

Instrumental power-seeking might be less dangerous if the self-model of the agent is large and includes individual humans, groups, or even all of humanity and if we can reliably shape it that way.

It is natural for humans to for form a self-model that is bounded by the body, though it is also common to be only the brain or the mind, and there are other self-models. See also Intuitive Self-Models [? · GW].

It is not clear what the self-model of an LLM agent would be. It could be

• the temporary state of the execution of the model (or models),
• the persistently running model and its memory state,
• the compute resources (CPU/GPU/RAM) allocated to run the model and its collection of support programs,
• the physical compute resources in some compute center(s),
• the compute center as an organizational structure that includes the staff to maintain and operate not only the machines but also the formal organization (after all, without that, the machines will eventually fail), or
• dito but including all the utilities and suppliers to continue to operate it.

There is not as clear a physical boundary as in the human case. But even in the human case, esp. babies depend on caregivers to a large degree.

There are indications that we can shape the self-model of LLMs: Self-Other Overlap: A Neglected Approach to AI Alignment [LW · GW]

I really don't like when people downvote so heavily without giving reasons - think this is nicely argued! 

One issue I do have is that Bob Fischer, the conductor of the Rethink study, warned about exactly what you are sorta doing here in being like ah now we can use x amount of shrimp and saying we can trolly problem a human for that many. This is just one contention, but I think the point is important and people willing to take weird/ controversial ideas seriously (especially here!) should take it more seriously! 

Yeah I just wanted to check that nobody is giving away money before I go do the exact opposite thing I've been doing

This sounds related to my complaint about the YUDKOWSKY + WOLFRAM ON AI RISK debate:

I wish there had been some effort to quantify @stephen_wolfram's "pockets or irreducibility" (section 1.2 & 4.2) because if we can prove that there aren't many or they are hard to find & exploit by ASI, then the risk might be lower.

I got this tweet wrong. I meant if pockets of irreducibility are common and non-pockets are rare and hard to find, then the risk from superhuman AI might be lower. I think Stephen Wolfram's intuition has merit but needs more analysis to be convicing.