LessWrong 2.0 Reader

I have an intuition like: Minds become less idiosyncratic as they grow up.

A couple of intuition pumps:

(1) If you pick a game, and look at novice players of that game, you will often find that they have rather different "play styles".  Maybe one player really likes fireballs and another really like crossbows.  Maybe one player takes a lot of risks and another plays it safe.

Then if you look at experts of that particular game, you will tend to find that their play has become much more similar.  I think "play style" is mostly the result of two things: (a) playing to your individual strengths, and (b) using your aesthetics as a tie-breaker when you can't tell which of two moves is better.  But as you become an expert, both of these things diminish: you become skilled at all areas of the game, and you also become able to discern even small differences in quality between two moves.  So your "play style" is gradually eroded and becomes less and less noticeable.

(2) Imagine if a society of 3-year-olds were somehow in the process of creating AI, and they debated whether their AI would show "kindness" to stuffed animals (as an inherent preference, rather than an instrumental tool for manipulating humans). I feel like the answer to this should be "lol no". Showing "kindness" to stuffed animals feels like something that humans correctly grow out of, as they grow up.

It seems plausible to me that something like "empathy for kittens" might be a higher-level version of this, that humans would also grow out of (just like they grow out of empathy for stuffed animals) if the humans grew up enough.

(Actually, I think most humans adults still have some empathy for stuffed animals. But I think most of us wouldn't endorse policies designed to help stuffed animals. I'm not sure exactly how to describe the relation that 3-year-olds have to stuffed animals but adults don't.)

I sincerely think caring about kittens makes a lot more sense than caring about stuffed animals. But I'm uncertain whether that means we'll hold onto it forever, or just that it takes more growing-up in order to grow out of it.

Paul frames this as "mostly a question about idiosyncrasies and inductive biases of minds rather than anything that can be settled by an appeal to selection dynamics." But I'm concerned that might be a bit like debating the odds of whether your newborn human will one day come to care for stuffed animals, instead of whether they will continue to care for them after growing up. It can be very likely that they will care for a while, and also very likely that they will stop.

I strongly suspect it is possible for minds to become quite a lot more grown-up than humans currently are.

(I think Habryka may have been saying something similar to this.)

Still, I notice that I'm doing a lot of hand-waving here and I lack a gears-based model of what "growing up" actually entails.

Good catch, thanks! Fixed now.

Why?

FYI I remember being vaguely dissatisfied wth the early exercises in the book, and recommend skipping ahead to somewhere in the middle of the first-half.

i'm enjoying this. going through the questions right now, might do all of them

had a notable experience with one of the early questions:

Thank you for writing this up! I've purchased some subscriptions and plan to block out time to play around with some of these services and get familiar with them.

criticism in LW comments is why he stopped writing Sequences posts

I wasn't aware of this and would like more information. Can anyone provide a source, or report their agreement or disagreement with the claim?

No. But I’m skeptical: seems hard to imagine provable safety, much less competitive with the default path to powerful AI, much less how post-hoc evals are relevant.

Out of curiosity, do you have any thoughts on the importance / feasibility of formal verification / mathematically "provable" safety based approaches in these evals you mention?

Note to readers; it is an obligatory warning on any post like this that you should not run random scripts downloaded from the internet without reading them to see what they do, because there are many harmful things they could be doing.