LessWrong 2.0 Reader

Your linked post on The Obliqueness Thesis [LW · GW] is curious. You conclude thus:

Obliqueness obviously leaves open the question of just how oblique. It's hard to even formulate a quantitative question here. I'd very intuitively and roughly guess that intelligence and values are 3 degrees off (that is, almost diagonal), but it's unclear what question I am even guessing the answer to. I'll leave formulating and answering the question as an open problem.

I agree, values and beliefs are oblique. The 3 spatial dimensions are also mutually oblique, as per General Relativity. A theory of obliqueness is meaningless if it cannot specify the angles [ I think in a correct general linear algebra, everything would be treated as [at least potentially] oblique to everything else, but that doesn't mean I refuse to ever treat the 3 spatial dimensions as mutually orthogonal ].

As with the 3 spatial dimensions in practical ballistics, with the value dimension and the belief dimension in practical AI alignment, there are domains of discussion where it is appropriate to account for the skew between the dimensions and domains where it is appropriate to simply treat them as orthogonal. Discussions of alignment theory such as the ones in which you seek to insert your Obliqueness thesis, are a domain in which the orthogonality assumption is appropriate. We cannot guess at the skew with any confidence in particular cases, and with respect to any particular pre-chosen utility/valence function term versus any particular belief-state [e.g. "what is the dollar value of a tulip?" x "is there a teapot circling Mars?"], the level of skew is almost certain to be negligible.

Planning for anthropically selected futures, on the other hand, is a domain where the skew between values and beliefs becomes relevant. There is less point reasoning in detail about pessimized futures or others that disagree with our values [such as the ones in which we are dead], no matter how likely or unlikely they might be "in a vacuum", if we're trying to hyperstition our way into futures we like. But this is an esoteric and controversial argument and not actually required to justify why I don't think it's useful to consider [sufficiently] strong AI as "what can eat the sun".

All that's required to justify why I don't think it's useful to consider [sufficiently] strong AI as "what can eat the sun", is that what you propose is a benchmark of capability, or intelligence. Benchmarks of intelligence [say, of bureaucrats or chimps] are not questions of fact. They are social fictions chosen for their usefulness. If we are dead in the vast, vast supermajority of the worlds where the benchmark would otherwise be useful - in this case, the worlds where people deploy an AI that they do not know, ahead of time, if it will or will not be strong enough to eat the Sun - it is not, particularly, a benchmark we should be etching into the wood, from our present standpoint.

But like, I wouldn't be surprised if, say, someone trained something that performed comparably to LLMs on a wide variety of benchmarks, using much less "data"... and then when you look into it, you find that what they were doing was taking activations of the LLMs and training the smaller guy on the activations. And I'll be like, come on, that's not the point; you could just as well have "trained" the smaller guy by copy-pasting the weights from the LLM and claimed "trained with 0 data!!". And you'll be like "but we met your criterion!" and I'll just be like "well whatever, it's obviously not relevant to the point I was making, and if you can't see that then why are we even having this conversation". (Or maybe you wouldn't do that, IDK, but this sort of thing--followed by being accused of "moving the goal posts"--is why this question feels frustrating to answer.)

Link?

One big reason I might expect an AI to do a bad job at alignment research is if it doesn't do a good job (according to humans) of resolving cases where humans are inconsistent or disagree. How do you detect this in string theory research? Part of the reason we know so much about physics is humans aren't that inconsistent about it and don't disagree that much. And if you go to sub-topics where humans do disagree, how do you judge its performance (because 'be very convincing to your operators' is an objective with a different kind of danger).

Another potential red flag is if the AI gives humans what they ask for even when that's 'dumb' according to some sophisticated understanding of human values. This could definitely show up in string theory research (note when some ideas suggest non-string-theory paradigms might be better, and push back on the humans if the humans try to ignore this), it's just intellectually difficult (maybe easier in loop quantum gravity research heyo gottem) and not as salient without the context of alignment and human values.

Robin says that we have less cultural diversity than in the past. I am not sure about that. In the past, we had geographically separated cultures, but within each culture, there wasn't enough space for many subcultures. Today, the cultures are closer, but the subcultures can be larger. Hundred years ago, there would be no such thing as the rationalist community. (Even using the example from Robin's article: it's not like Amish are living on some distant island.)

I don't understand the argument why colonizing the stars would not fix the problem (of cultural drift leading to low fertility). My worry would be the opposite -- that the future will belong to those who replicate the fastest (and sacrifice everything else for that goal).

(Yeah, you responded, but felt not that operationalized and seemed doable to flesh out as you did.)

GoodFire is now available for use, and it's easy and fun to use! You should check it out if you're interested in studying why LLMs do the things they do!

The characters don't live in a world where sharing or smoothing risk is already seen as a consensus-valuable pursuit; thus, they will have to be convinced by other means.

I gave their world weirdly advanced [from our historical perspective] game theory to make it easier for them to talk about the question.

isn't the fact fluoride in toothpaste and brushing twice daily common likely to make it so there wouldn't be any dental harm from non-fluridated water? I've not done a deep dive on fluoride but my rough thinking is (a) it's possible it has harm (b) most people use fluoride/xylitol in toothpaste so the benefits of fluoride in water supplies seems not only negligible but likely non-existent in this day and age  

Yeah, that makes sense, especially if combined with the feature that allows users to disagree with specific parts of the post, as Michael notes [LW(p) · GW(p)]. (Though note that the disagree vote is anonymous, whereas disagreeing with a selection is public, so the two aren’t fully comparable.)