LessWrong 2.0 Reader

I don't think I agree with the "but only one thought can be there at a time" part.

I’m probably just defining “thought” more broadly than you. The cortex has many areas. The auditory parts can be doing some auditory thing, and simultaneously the motor parts can be doing some motor thing, and all that together constitutes (what I call) a “thought”.

I don’t think anything in the series really hinges on the details here. “Conscious awareness” is not conceptualized as some atomic concept where there’s nothing else to say about it. If you ask people to describe their conscious awareness, they can go on and on for hours. My claim is that: when they go on and on for hours describing “conscious awareness”, all the rich details that they’re describing can be mapped onto accurate claims about properties of the cortex and its activation states. (E.g. the next part of this comment.)

I think each of the workspaces have their own short memory which spans maybe like 2seconds.

I agree that, if some part of the cortex is in activation state A at time T, those particular neurons (that constitute A) will get less and less active over the course of a second or two, such that at time T+1 second, it’s still possible for other parts of the cortex to reactivate activation state A via an appropriate query. I don’t think all traces of A immediately disappear entirely every time a new activation state appears.

Again, I think the capability of the cortex to have more than one incompatible generative models active simultaneously is extremely limited, and that for most purposes we should think of it as only having a single (MAP estimate) generative model active. But the capability to track multiple incompatible models simultaneously does exist, to some extent. I think this slow-fading thing is one application of that capability. Another is the time-extended probabilistic inference thing that I talked about in §2.3.

Thanks Ben!

Great catch on the discrepancy between the podcast and this post—the description on the podcast is a variant we experimented with, but what is published here is self-contained and accurately describes what was done in the original paper.

On generalization and unlearning/forgetting: averaging activations across two instances of a network might give the impression of inducing "forgetting," but the goal is not exactly unlearning in the traditional sense. The method aims to align certain parts of the network's latent space related to self and other representations without losing useful distinctions. However, I agree with your point about the challenge of generalization using this set-up: this proof of concept is a more targeted technique that benefits from knowing the specific behaviors you want to discourage (or mitigate), rather than yet being a broad, catch-all solution. 

And good points on the terminology around "self," "deception," and "other:" these simple models clearly don't have intentionality or a sense of self like humans do, so these terms are being used more as functional stand-ins than precise descriptions. Will aim to keep this clearer as we continue communicating about this work.

Would also love to hear more about your work on empathy and the amygdala as it relates to alignment. Thanks again for the great feedback!

OK, my theory is:

• (A) There’s a thing where people act kind towards other people because it’s in their self-interest to act kind—acting kind will ultimately lead to eating yummier food, avoiding pain, and so on. In everyday life, we tend to associate this with flattery, sucking up, deception, insincerity, etc., and we view it with great skepticism, because we recognize (correctly) that such a person will act kind but then turn right around and stab you in the back as soon as the situation changes.
• (B) There’s a separate thing where people act kind towards other people because there’s some innate drive / primary reward / social instinct closely related to acting kind towards other people, e.g. feeling that the other person’s happiness is its own intrinsic reward. In everyday life, we view this thing very positively, because we recognize that such a person won’t stab you in the back when the situation changes.

I keep trying to pattern-match what you’re saying to:

• (C) [which I don’t believe in] This is a third category of situations where people are kind. Like (A), it ultimately stems from self-interest. But like (B), it does not entail the person stabbing you in the back as soon as the situation changes (such that stabbing you in the back is in their self-interest). And the way that works is over-generalization. In this story, the person finds that it’s in their self-interest to act kind, and over-generalizes this habit to act kind even in situations where it’s not in their self-interest.

And then I was saying that that kind of over-generalization story proves too much, because it would suggest that I would retain my childhood habit of not-driving-cars, and my childhood habit of saying that my street address is 18 Main St., etc. And likewise, it would say that I would continue to wear winter coats when I travel to the tropics, and that if somebody puts a toy train on my plate at lunchtime I would just go right ahead and eat it, etc. We adults are not so stupid as to over-generalize like that. We learn to adapt our behavior to the situation, and to anticipate relevant consequences.

But maybe that’s not what you’re arguing? I’m still kinda confused. You wrote “But across almost all environments, you get positive feedback from being nice to people and thus feel or predict positive valence about these.” I want to translate that as: “All this talk of stabbing people in the back is irrelevant, because there is practically never a situation where it’s in somebody’s self-interest to act unkind and stab someone in the back. So (A) is really just fine!” I don’t think you’d endorse that, right? But it is a possible position—I tend to associate it with @Matthew Barnett [LW · GW]. I agree that we should all keep in mind that it’s very possible for people to act kind for self-interested reasons. But I strongly don’t believe that (A) is sufficient for Safe & Beneficial AGI. But I think that you’re already in agreement with me about that, right?

pollsters will have attempted to correct mistakes, and if they knew that there would be an R/D bias this time, they'd adjust in the opposite way, hence the error must be unpredictable.

Exactly. Silver has discussed this dynamic in some of his old FiveThirtyEight articles. The key is to appreciate that polling error is not an effect one can naively predict by looking at past data, because it is mediated by polling agencies’ attempts to correct it.

• Correspondingly, in the actual brain algorithm (“territory”), any possible thought can be represented by the cortex (by definition), but only one thought can be there at a time (to a first approximation, see §2.3 below).

I don't think I agree with the "but only one thought can be there at a time" part.

I think (but not at all sure) that different (sensory) modalities can process (in unfocused states) unrelated information/thoughts simultaneously.

Here's my current rough model that might be wrong in many ways:

I think each of the workspaces have their own short memory which spans maybe like 2seconds. (E.g. sometimes when I become aware of an earworm I feel like it must've been there for at least 2 seconds already even though I previously wasn't aware of it.)

(I currently don't know whether there is a "space of awareness" or whether (as in the image above) it's just that some workspace is broadcasting its contents broadly.)

(I think it sometimes happened to me that I was daydreaming about something which was written to memory (although not very strongly) but where I didn't notice that I was thinking that and only later realized "oh wait I thought about that before". So perhaps sth was only weakly attended on and left some trace in memory but then either the "produce self-reflective thought" or the "associate self-reflective thought with homunculus" step didn't happen. But idk I'd want to re-observe this phenomenon to be sure it's not just fabricated. EDIT: Actually from having made a break and daydreamed a bit now, I think it rather is the case that usually one decently often has a self-reflective thought (about what object-level thoughts one thinks) attended on so there's also a memory of oneself thinking about sth. However sometimes in deep daydreams (or other states like flow) self-reflective thoughts are only very rarely attended on and sometimes never. I think often I notice roughly about what I was daydreaming afterwards, but maybe sometimes the daydream is disrupted by something outside so that then there wasn't a self-reflective thought written to memory, which might then later make me slightly surprised when I notice that I e.g. daydreamed about a particular movie earlier but didn't notice that at the time. Aka: I think the self-reflective thoughts still get produced unconsciously when daydreaming, but they might not get attended on so no self-associated memory of the event is formed.)

Fair enough. I still think it aligns well with the overall vibe and ethos of the setting.

I think in many cases this is a feature rather than a bug, I'd be curious as to whether you had an example of a time when having made the prediction would be deleterious to realising the desired outcome? Assuming as Adam says that you're more interested in the outcome than the prediction being accurate.

I don't know if it is a mismatch, actually. Many things are rated highly but still underrated, and vice versa.

I've only skimmed this post, but I like it because I think it puts into words a fairly common believed model (that I disagree with). I've heard "it's all just a stack of heuristics" as an explanation of neural networks and as a claim that all intelligence is this, from several people. (Probably I'm overinterpreting other people's words to some extent, they probably meant a weaker/nuanced version. But like you say, it can be useful to talk about the strong version).

I think you've correctly identified the flaw in this idea (it isn't predictive, it's unfalsifiable, so it isn't actually explaining anything even if it feels like it is). You don't seem to think this is a fatal flaw. Why?

You seem to answer

However, the key interpretability-related claim is that heuristics based decompositions will be human-understandable, which is a more falsifiable claim.

But I don't see why heuristics based decompositions will be human-understandable is an implication of the theory. As an extreme counterexample, logic gates are interpretable, but when stacked up into a computer they are ~uninterpretable. It looks to me like you've just tacked an interpretability hypothesis onto a heuristics hypothesis.

I was under the impression that “illusions of free will” was a standard term in the literature, but I just double-checked and I guess I was wrong, it just happened to be in one paper I read. Oops. So I guess I’m entitled to use whatever term seems best to me.

I mildly disagree that it’s unrelated to “free will”, but I agree that all things considered your suggestion “illusions of intentionality” is a bit better. I’m changing it, thanks!