LessWrong 2.0 Reader

Thanks for the post! Frankly this is a sub-field of alignment that I have not been following closely, so it is very useful to have a high-level comparative overview.

I have a question about your thoughts on what 'myopia verification' means in practice.

Do you see 'myopia' as a single well-defined mathematical property that might be mechanically verified by an algorithm that tests the agent? Or is it a more general bucket term that means `bad in a particular way', where a human might conclude, based on some gut feeling when seeing the output of a transparency tool, that the agent might not be sufficiently myopic?

What informs this question is that I can't really tell when I re-read your Towards a mechanistic understanding of corrigibility [LW · GW] and the comments there. So I am wondering about your latest thinking.

I translated what the median U.S.A. teacher has into generic language to come up with the tutor criteria. That is based on anecdotal data from observing n of approx. 50 and hearing first-hand of another n of approx. 100.

Yes, in the "good" school districts in the US that are big enough, there can be classes like Theater, Biotechnology, a full spread of AP/IRB classes, Computer Programming back in the '90s, a school newspaper / tv station, an actual Film (ie movie creation) class, etc.

Free university lectures are good (and a step up from the widely available YouTube classes, for the in-person Q&A with other students), but in many cases there is physical equipment (science labs, performance venues, machine shops) that wouldn't be available for free.

For a typical kid with high self-directedness, you probably only need 10 hr/week of professional guidance, plus continuing to have involved parents. The key point (I'm synthesizing this from How Children Learn and How Children Fail, by John Holt) is that the learner needs two things to succeed:

1. Access to a trusted authority, to get their questions answered (eg "how do I even get started with this?" or "Did I do this right?" or "What is this one tricky word?")

2. Someone observing them while they work, to spot the unknown unknowns and answer the questions the student doesn't even know they have.

https://www.lesswrong.com/posts/WBw8dDkAWohFjWQSk/the-cluster-structure-of-thingspace [LW · GW]

Words point to clusters of similar things. If we just want to point in the rough direction of a cluster, use a general term like "AGI", if you want to talk about a more well defined and specific set, you can say things like "an AI that can do all tasks related to AI research at least as well as a top human researcher".

Also, words are pointing to empirical clusters, so fictional examples might not be in any of the real world clusters. Real world animals are clustered into mammal, bird, reptile ect. But some fictional animals like gryphons don't fit in that classification.

There’s a thing where entrepreneurs are optimistic. Sometimes ridiculously so. This is because entrepreneurship requires a lot of optimism to think that what you’re doing can work. But that optimism can lead you astray and cause you to work on things that are too hard.

Jess and Robin will never be entrepreneurs. They aren’t optimistic enough. But Elon may be over-optimistic. The sheer scope of the project makes it hard to guess accurately what the difficulties are going to be without getting pretty far into trying it and seeing where you get stuck.

And I’m just thinking about getting a Mars colony at all. I do think “self-sustaining” is a ridiculously high bar, much higher than simply having some people living on Mars. But how important is “self-sustaining” as a short-term goal? — if the colony works at all, there will be a path where the amount of Earth resources required to sustain it shrinks over time due to market forces.

In any reasonably large space of possibilities, the actual optimum is usually really weird. There is no sharp line between really healthy cooking; biochemical manufacture of medicine; and bootstraping medical nanotech. If a fruit gnome has a dangerous or unhealthy job, and would quit that job if they could afford it, does spelling out next weeks lottery numbers in some pattern or code that the gnome will understand count as a healthy diet? Optimums are weird things.

I don't know the US situation firsthand, but it seems like it could get worse toward the election. Maybe move to Europe?

I saw this "stopped clock" assumption catching a bunch of people with COVID-19, so I wrote a quick post [LW · GW] on why it seems unlikely to be a good strategy.

Not sure. There are other differences, too. – In the way the keys respond, and the fact that on a piano key you have much more freedom in choosing a convenient place for your finger. – This could all lead to less crampiness. So again, talking to an expert is the best bet. I've had good experiences with online PT consultations for weightlifting-related aches and pains.

Entirely fair. Nothing done, then.

Notice that I didn't use the term agent, because I personally believe that goal-directedness and agency are distinct (though probably linked). So I agree with you're intuition that an agent should probably know its goal, but I disagree with the proposal that it must do so to be goal-directed towards that goal.

That being said, I do agree that there is a difference in kind between the result of RL and an optimizer when following a goal. One way we (Joe Collman, Michele Campolo, Sabrina Tang and I) think about it is through the "source of directedness": self-directed (like an optimizer), hardcoded in a direction (like an optimized system), or even self-directed with some constraint/initial direction (mesa-optimizer, that is an optimized optimizer).

they just execute with no sense of purpose, even if their pattern of behavior is well predicted by modeling them as if they had goals.

On that part I agree with Dennett that the definition of having a goal is for the pattern of behavior to be well predicted by modeling the system as having goals (taking the intentional stance). It seems you disagree, but maybe this boils down to the separation between goal-directedness and agency I pointed above.