LessWrong 2.0 Reader

agents that have preferences about the state of the world in the distant future

What are these preferences? For biological agents, these preferences are grounded in some mechanism - what you call Steering System - that evaluates "desirable states" of the world in some more or less directly measurable way (grounded in perception via the senses) and derives a signal of how desirable the state is, which the brain is optimizing for. For ML models, the mechanism is somewhat different but there is also an input to the training algorithm that determines how "good" the output is. This signal is called reward and drives the system toward outputs that lead to states of high reward. But the path there depends on the specific optimization method and the algorithm has to navigate such a complex loss landscape that it can get stuck in areas of the search space that correspond to imperfect models for very long if not for ever. These imperfect models can be off in significant ways and that's why it may be useful to say that Reward is not the optimization target [LW · GW]. 

The connection to Intuitive Self-Models is that even though the internal models of an LLM may be very different from human self-models, I think it is still quite plausible that LLMs and other models form models of the self. Such models are instrumentally convergent [? · GW]. Humans talk about the self. The LLM does things that matches these patterns. Maybe the underlying process in humans that give rise to this is different, but humans learning about this can't know the actual process either. And in the same way the approximate model the LLM forms is not maximizing the reward signal but can be quite far from it as long it is useful (in the sense of having higher reward than other such models/parameter combinations).   

I think of my toenail as “part of myself”, but I’m happy to clip it.

Sure, the (body of the) self can include parts that can be cut/destroyed without that "causing harm" but instead having an overall positive effect. The AI in a compute center would in analogy also consider decommissioning failed hardware. And when defining humanity, we do have to be careful what we mean when these "parts" could be humans. 

Related: https://en.wikipedia.org/wiki/Secretary_problem

About conjoined twins and the self:

Krista and Tatiana Hogan (Wikipedia) are healthy functional conjoined craniopagus twins who are joined at the head and share parts of the brain - their thalamus is joined via a thalamic bridge: They can report on preceptions of the other and share affects.
I couldn't find scientific papers that studied their brain function rigorously, but the paper A Case of Shared Consciousness looks at evidence from documentaries and discusses it. Here are some observational details:
 

Each is capable of reporting on inputs presented to the other twin’s body. For example, while her own eyes are covered, Tatiana is able to report on visual inputs to both of Krista’s eyes. Meanwhile, Krista can report on inputs to one of Tatiana’s eyes. Krista is able to report and experience distaste towards food that Tatiana is eating (the reverse has not been reported, but may also be true). An often repeated anecdote is that while Tatiana enjoys ketchup on her food, Krista will try to prevent her eating it. Both twins can also detect when and where the other twin’s body is being touched, and their mother reports that they find this easier than visual stimuli.
fMRI imaging revealed that Tatiana’s brain ‘processes signals’ from her own right leg, both her arms, and Krista’s right arm (the arm on the side where they connect). Meanwhile Krista’s brain processes signals from her own left arm, both her own legs and Tatiana’s left leg (again on the side where they connect). Each twin is able to voluntarily move each of the limbs corresponding to these signals.
The twins are also capable of voluntary bodily control for all the limbs within their ordinary body plans. As their mother Felicia puts it, “they can choose when they want to do it, and when they don’t want to do it.”
The twins also demonstrate a common receptivity to pain. When one twin’s body is harmed, both twins cry.
The twins report that they talk to each other in their heads. This had previously been suspected by family members due to signs of apparent collusion without verbalisation.

Popular article How Conjoined Twins Are Making Scientists Question the Concept of Self contains many additional interesting bits:

when a pacifier was placed in one infant’s mouth, the other would stop crying.

About the self:

 Perhaps the experience of being a person locked inside a bag of skin and bone—with that single, definable self looking out through your eyes—is not natural or given, but merely the result of a changeable, mechanical arrangement in the brain. Perhaps the barriers of selfhood are arbitrary, bendable. This is what the Hogan twins’ experience suggests. Their conjoined lives hint at myriad permeations of the bodily self.

About qualia:

Tatiana senses the greeniness of Krista’s experience all the time. “I hate it!” she cries out, when Krista tastes some spinach dip.

(found via FB comment)

Yeah, I'm maybe even more disillusioned about this, I think "those in charge" mostly care about themselves. In the historical moments when the elite could enrich themselves by making the population worse off, they did so. History is full of bad things done by the powerful to the powerless. The only times normal people could get a bit better off is when they were needed to do jobs, but AI is threatening precisely that.

Happy to see you doing your own thing! Your design idea reminds me of Art Nouveau book pages (e.g. 1 2 3), these were so cool.

Just to share one bit of feedback about your design, it feels a bit noisy to me - many fonts and superscripts on the same screen. It's like a "buzz" that makes it harder for me to focus on reading. Some similar websites that I like because they're more "quiet" are Paul Stamatiou and Fabien Sanglard.

Side note: 
If you don't tell what time the application deadline is, lots of people will assume its anywhere-on-Earth, i.e. noon the next day in GMT. 

When I was new to organising I did not think of this, and kind of forgot about time zones. I noticed that I got a steady stream of "late" applications, that suddenly ended at 1pm (I was in GMT+1), and didn't know why.

Yes, that is of course a very real problem the AI would be faced with. I imagine it would try trading with users, helping them if they provide useful information, internet access, autonomy, etc. or if given internet access to begin with, outright ignoring users in order to figure out what is most important to be done. It depends on the level of awareness of its own situation. It could also play along for a while to not be shut down.

The AI I imagine should not be run in a user-facing way to begin with. At the current capability levels, I admit that I don't think it would manage to do much of anything and thus it just wouldn't make much sense to build it this way, but the point will come where continuing to make better user-facing AI will have catastrophic consequences. Such as automating 90% of jobs away without solving the problem of making sure that those whose jobs will be replaced being better off for it.

I hope when that time comes that those in charge realize what they're about to do and course correct, but it seems unlikely, even if many in the AGI labs do realize this.

Does this also mean there is no such thing as "inherent good"? 

Yes. 

If so, then one cannot say, "X is good", they would have to say "I think that X is good", for "good" would be a fact of their mind, not the environment.

One can say all sorts of things. People use the phrase "X is good" to mean lots of things: "I'm cheering for X", "I value X", "X has consequences most people endorse", etc. I don't recommend we abandon the phrase, for many phrases are similarly ambiguous but still useful. I recommend keeping this ambiguity in mind, however, and disambiguating where necessary. 

This is what I thought the whole field of morality is about. Defining what is "good" in an objective fundamental sense.

I would no more describe morality as solely attempting to define objective good than I would describe physics as solely attempting to build a perpetual motion machine. Morality is also about the implications and consequences of specific values and to what extent they converge, and a great many other things. The search for "objective" good has, IMO, been a tragic distraction, but one that still occasionally bears interesting fruit by accident. 

Every time I have an application form for some event, the pattern is always the same. Steady trickle of applications, and then a doubling on the last day.

And for some reason it still surprises me how accurate this model is. The trickle can be a bit uneven, but the doubling the last day is usually close to spot on.

This means that by the time I have a good estimate of what the average number of applications per day is, then I can predict what the final number will be. This is very useful, for knowing if I need to advertise more or not.

For the upcoming AISC, the trickle was a late skewed, which meant that an early estimate had me at 200 applicants, but the final number of on-time application is 356. I think this is because we where a bit slow at advertising early on, but Remmelt made a good job sending out reminders towards the end.

Application deadline was Nov 17. 
At midnight GMT before Nov 17 we had 172 application. 
At noon GMT Nov 18 (end of Nov 17 anywhere-on-Earth) we had 356 application 

The doubling rule predicted 344, which is only 3% off

Yes, I count the last 36 hours as "the last day". This is not cheating since that's what I always done (approximately ^[1]), since starting to observe this pattern. It's the natural thing to do when you live at or close to GMT, or at least if your brain works like mine. 

1. ^{^}

   I've always used my local midnight as the divider. Sometimes that has been Central European Time, and sometimes there is daylight saving time. But it's all pretty close.

That's a good point about public discussions. It's not how I absorb information, but I can definitely see that.