LessWrong 2.0 Reader

A tiny case of this I wrote about long ago: https://markxu.com/stop-asking-people-to-maximize

Something someone technical and interested in forecasting should look into:  can LLMs reliably convert peoples claims into a % of confidence through sentiment analysis? This would be useful for Forecasters I believe (and rationality in general)

I can imagine this growing into the default reference that people use when talking about whether labs are behaving responsibly.

I hope that this resource is used as a measure of relative responsibleness, and this doesn't get mixed up with absolute responsibleness. My understanding is that the resource essentially says, "here's some things that would be good– let's see how the labs compare on each dimension." The resource is not saying "if a lab gets a score above X% on each metric, then we are quite confident that the lab will not cause an existential catastrophe."

Moreover, my understanding is that the resource is not taking a position on whether or not it is "responsible"– in some absolute sense– for a lab to be scaling toward AGI in our current world. I see the resource as saying "conditional on a lab scaling toward AGI, are they doing so in a way that is relatively more/less responsible compared to the others that are scaling toward AGI."

This might be a pedantic point, but I think it's an important one to emphasize– a lab can score in 1st place and still present a risk to humanity that reasonable people would still deem unacceptable & irresponsible. 

I think we can more easily and generally justify the use of the intentional stance. Intentionality requires only the existence of some process (a subject) that can be said to regard things (objects). We can get this in any system that accepts input and interprets that input to generate a signal that distinguishes between object and not object (or for continuous "objects", more or less object).

For example, almost any sensor in a circuit makes the system intentional. Wire together a thermometer and a light that turns on when the temperature is over 0 degrees, off when below, and we have a system that is intentional about freezing temperatures.

Such a cybernetic argument, to me at least, is more appealing because it gets down to base reality immediately and avoid the need to sort out things people often want to lump in with intentionality, like consciousness.

Possibly. But there is no optimization pressure from pre-training on the relationship between MLPs and transcoders. The MLPs are the thing that pre-training optimizes (as the "full-precision" master model), while transcoders only need to be maintained to remain in sync with the MLPs

I see. I was in fact misunderstanding this detail in your training setup. In this case, only engineering considerations really remain: these boil down to incorporating multiple transcoders simultaneously and modeling shifting MLP behavior with transcoders. These seem like tractable, although probably nontrivial and, because of the LLM pretraining objective, quite computationally expensive. If transcoders catch on, I hope to see someone with the compute budget for it run this experiment!

Secondly, following Dennett, the point of modeling cognitive systems according to the intentional stance is that we evaluate them on a behavioral basis and that is all there is to evaluate.

I am confused on this point. Several people have stated that Dennett believes something like this, e.g., Quintin and Nora argue [LW · GW] that Dennett is a goal "reductionist," by which I think they mean something like "goal is the word we use to refer to certain patterns of behavior, but it's not more fundamental than that."

But I don't think Dennett believes this. He's pretty critical of behaviorism, for instance, and his essay Skinner Skinned does a good job, imo, of showing why this orientation is misguided. Dennett believes, I think, that things like "goals," "beliefs," "desires," etc. do exist, just that we haven't found the mechanistic or scientific explanation of them yet. But he doesn't think that explanations of intention will necessarily bottom out in just their outward behavior, he expects such explanations to make reference to internal states as well. Dennett is a materialist, so of course at the end of the day all explanations will be in terms of behavior (inward or outward), on some level, much like any physical explanation is. But that's a pretty different claim from "mental states do not exist." 

I'm also not sure if you're making that claim here or not, but curious if you disagree with the above? 

That seems fair enough!

(I would agree-react but I can't actually make it)

Thanks :) the recalibration may take a while… my intuition is still fighting ;)

Thank you for this, I'm conducting user interviews right now and there were some suprising things in your review, as well as obviously good ideas that I would probably have missed. Organizing meetups in the field would not have occured to me, and is a good idea.