LessWrong 2.0 Reader

In retrospect, I've become more optimistic about alignment, and I now think there's quite a bit more evidence towards the low-stakes version of the alignment problem being quite easy to solve for AIs, and while I don't expect anything close to formal proof (because of edge cases), I now think that the techniques that were developed are enough such that we can declare victory on the low-stakes alignment problem, and as a consequence I now think we can declare the outer alignment problem to be mostly solved at this point (given Paul's definition of low-stakes alignment).

The fact that a lot of AI risk scenarios rely more on deceptively aligned AI changing the world in an abrupt and irreversible way than in the past corroborates this.

I reject the principled distinction [LW · GW]. To me, it's more of a spectrum.

I'm thinking the causal closure part is more about the soul not existing than about anything else.

Causal closure is impossible for essentially every interesting system, including classical computers (my laptop currently has a wiring problem that definitely affects its behavior despite not being the sort of thing anyone would include in an abstract model).

Are there any measures of approximate simulation that you think are useful here? Computer science and nonlinear dynamics probably have some.

Love the drawings! I feel like more drawings would make this community easier to understand to outsiders.

I'm guilty of n°3 when people defend some hype products while saying untrue things about how they are better than the competition. Obvious example about Apple's products "doing something no other phone does" when the functionality in question was introduced 7 years ago on Android. Not exactly the same since they do not frame it as preference but try to rationalize it but I'll remember this next time to be nicer.

Excellent points! Yes, this is definitely a fun and interesting thing to engage with intellectually so long as both people feel like it's being done in a non-judgmental or agendic way. Part of why I included the paragraphs about non-filtering being hard for some people is that I know there are some brains for which this genuinely doesn't feel like it "should" be hostile or pressurey, since they don't perceive it that way... but as in all things, that's why your point about actually paying attention to what the other person says and taking it seriously is so important.

karma should also transfer automatically

Especially if it's something as non-committal as "this mechanism could maybe matter". Does that really invalidate the neuron doctrine?

I agree each of the "mechanisms that maybe matter" are tenuous by themselves, the argument I'm trying to make here is hits-based. There are so many mechanisms that maybe matter, the chances of one of them mattering in a relevant way is quite high.

there's imo probably not any (even-nearly-implementable) ceiling for basically any rich (thinking-)skill at all^[1] — no cognitive system will ever be well-thought-of as getting close to a ceiling at such a skill — it's always possible to do any rich skill very much better (I mean these things for finite minds in general, but also when restricting the scope to current humans)

(that said, (1) of course, it is common for people to become better at particular skills up to some time and to become worse later, but i think this has nothing to do with having reached some principled ceiling; (2) also, we could perhaps eg try to talk about 'the artifact that takes at most $n$ bits to specify (in some specification-language) which figures out $x$ units of math the quickest (for some $x$ sufficiently large compared to $n$)', but even if we could make sense of that, it wouldn't be right to think of it as being at some math skill ceiling to begin with, because it will probably very quickly change very much about its thinking (i.e. reprogram itself, imo plausibly indefinitely many times, including indefinitely many times in important ways, until the heat death of the universe or whatever); (3) i admit that there can be some purposes for which there is an appropriate way to measure goodness at some rich skill with a score in $[0,1]$, and for such a purpose potential goodness at even a rich skill is of course appropriate to consider bounded and optimal performance might be rightly said to be approachable, but this somehow feels not-that-relevant in the present context)

But I don't think these came about through training on synthetic thought-annotated texts.