Unrolling social metacognition: Three levels of meta are not enough.

I've curated this post, for these three reasons (starting with the most important):

• It lays out the implicit goings-on in a great many social interactions, and it does so with a simple, technical explanation.
• The main point is not one I've seen written up explicitly before, and seems to fit in well with my models of the world. It personally gave me a much greater affordance to notice when some social interaction / conflict is not has more than three meta levels to it, and requires the use of external working memory [LW · GW].
• It was quite readable and relatively short. It especially laid out useful metadata on the context - I feel I have a good sense of why you wrote the post and what caused you to believe the content of the post - without getting in the way of the core point of the post.

My thoughts on extensions and improvements:

• I'd be interested to read more concrete accounts of people using this when resolving interpersonal conflicts/confusions in their lives, and what particular heuristics helped them.
• Seeing people figure out how it affects the points+conclusions of other writings on this topic (e.g. the ones you link to by Duncan and myself). (Alas, I have opinions on this that do not fit in this small margin.)

---

Personally, this post managed to describe levels of social meta in common knowledge that I had run away scared from trying to explain in my post on the topic (I just quoted someone else's explanation at length). I spent a while trying to think about how to explain its involvement in one-on-one social interactions, and had not come up with anything as concrete and clear (if a little cognitively taxing) as this one.

Also, you mention that you think that the community probably would prefer this quick post than no post at all, and that you think others will be able and happy to think about it + test it for themselves if it seems important. This seems correct to me - when you only have the time for a two-hour post, writing it and giving this context at the head of a post is great, can be very valuable, and I'm really glad you did it. Generally it seems both that folks are very happy you wrote this MVP version, and that they will try to test it out. (To those of you who do try it out, writing your experiences up in a LW post can be really helpful to others!)

I’m intrigued by the explicit unrolling in contrast to circling. I wonder how much circling is an instance of developing overpowered tools on weird partly-orthogonal dimensions (like embodiment) because you haven’t yet discovered the basic simple structure of the domain.

• Like, a person might have a bunch of cobbled together hacks and heuristics (including things about narrative, and chunking next actions, and discipline) for maintaining their productivity. But a crisp understanding of the relevant psychology makes “maintaining productivity” a simple and mostly effortless thing to do.
• Or a person who spends years doing complicated math without paper. They will discover all kinds of tricks for doing mental computation, and they might get really good at these tricks, and building that skill might even have benefits in other domains. But at the end of the day, all of that training is blown out of the water as soon as they have paper. Paper makes the thing they were training hard to do, easy.

To what extent is Circling working hard to train capacities that are being used as workarounds for limited working memory and insufficient theoretical understanding the structure of human interaction?

(This is a real question. My guess is, “some, but less than 30%”.)

A lot of my strategies for dealing with situations of this sort are circling-y, and I feel like a lot of that is superfluous. If I had a better theoretical understanding, I could do the thing with much more efficiency.

For instance, I exert a lot of effort to be attuned to the other person in general and to be picking up subtle signs from them, and tracking where they’re at. If had a more correct theoretical understanding, a better ontology, I would only need to be tracking the few things that it turns out are actually relevant.

Since humans don’t know what those factors are, now, people are skilled at this sort of interaction insofar as they can track everything that’s happening with the other person, and as a result, also capture the few things that are relevant to the underlying structure.

I suspect that others disagree strongly with me here.

(Crossposted from here)

Alex felt it was bad that Alex felt that Bailey felt that Alex leaving out the milk was bad.

I want to point out that not all instances of the word "felt" mean the same thing here, and I want to split the two meanings into new words for clarity. I think this has consequences about what it means to have feelings about one's own feelings.

Bailey felt that Alex leaving out the milk was bad.

In this case, "feeling" stands for "evaluating". Bailey gives a low value to Alex's action, and in some sense, maybe even to Alex. E.g. Alex is dumb, or Alex's actions are suboptimal.

Alex felt that Bailey felt that Alex leaving out the milk was bad.

In this case, the top level "feeling" stands for "sensing". This is a complex phenomenon:

• X happens inside someone's mind;
• the person holding X produces some observable effect Y (e.g. they make a facial expression);
• the part of your brain specialized in this kind of work observes Y, infers X from it (you can't observe X!), and produces a conscious sensation Z (e.g. feeling judged);
• you consciously observe Z in yourself.

Alex felt it was bad that Alex felt that Bailey felt that Alex leaving out the milk was bad.

The top level "feeling" stands again for "evaluating". After observing Z in yourself, you give it a low value. But Z is a proxy for X given that X is true (you may be wrong inferring X from Y). Giving a low value to Z roughly means "I wish it was not the case that X", but it's difficult to recognize as such because I think it happens very rapidly and as a complex mental sensation rather than a sentence in your head. I posit that all feelings about one's own feelings are of this kind, and I'm interested in hearing counterexamples.

So, to recap:

Alex evaluated it was bad that Alex sensed that Bailey evaluated that Alex leaving out the milk was bad.

I think the unrolling is clearer this way.

A lot of excellent points in this post. I particularly like the one about having feelings about one's own feelings, which is something I think I've always understood on some level but perhaps the first time I thought about it consciously was from Seth Rogan's line in 40-Year-Old Virgin: "[Y]our depression is boring me for one thing, and it's actually making me a little depressed, which is then in turn making me more depressed that you're actually affecting my mood."

I think posts like this prove a more general point: we humans are thinking on many levels of meta, all the time, and insistence on pulling the levels apart to examine separately isn't "complicating things"; it's just a way of seeing more clearly what's already in our line of vision.

Thanks for the read (honestly, noticed some very interesting points IMO) but I kind of fail to understand what exactly is your claim about the method you introduced.

Are you saying that it is a good model representation of social interaction? If so I would partially agree. Its cool that the model captures all the mental steps all the participants are making (if you bother to completely unroll everything), but it's not computationally superior to saying that: things like calling someone "a downer" are general beliefs that rely on a varying empirical basis, and should be checked for their verity.

In other words - instead of saying that "Bailey is a downer" means "Alex felt it was bad that Alex felt that Bailey felt that Alex doing X was bad, for multiple values of X" you can say that it means "Alex believes he observed Bailey to act as 'a downer' a sufficient amount of times" and unroll that (ask Alex why does he think Bailey's a downer, what does being a downer mean or what does he base her being one on) only if necessary. Most people would understand this explanation more intuitively, in my experience. Some might even find it trivial, cause, for example, if you call someone irresponsible more than once, most of the times you are aware that you think he has acted "irresponsibly" a sufficient amount of times, even if you don't phrase it that exact way. And this explanation is not inferior to yours in the theoretical sense, it doesn't supply less data, and it does seem a little more cost effective, explanation-wise.

Are you saying that it is a good method for practical conflict resolution? It very well may be, but your experience only teaches us that after engaging in a cooperative activity with Alex for a while, he understood that he felt judged and was more inclined to believe you weren't judging him. Psychologically, engaging in a safe activity with someone, even your captor in a hypothetical hostage situation, will diffuse tension and humanize conflicting parties in each other's eyes. It could be that you could have played a short card game with Alex and he would have been more cooperative afterwards all the same.

Even if we claim that analyzing the problem is emotionally helpful in itself, the analysis doesn't have to be all that rigorous, coherent and complete. Many conflict resolution therapy methods focus on giving all parties in a conflict the opportunity to feel heard, which makes it easier to reach emotional catharsis, and therefore agreement. But feeling heard is only a equal to or lesser amount of understanding than being completely, logically understood. Therapy methods such as these may (or may not) be more effective in achieving results, or may be just less effort-demanding.

In order to establish that this method is particularly effective, we either have to get some experimental data showing it gets better outcomes in some criteria, or explain what it has going for it that other methods unequivocally don't.

Anyway, I enjoyed reading this and would be glad if you posted some of your refined conclusions in the future :)

Thank you so much for sharing this concept with me a few years ago, it's still in the top ten of largest coherent boosts another human gave me.

I want to mention, for citation reasons - I am pretty sure a large amount of the discussion about this topic as an explicit, defineable human-common-knowledge came into my friend group through you, despite that I and others have written posts using the concept since you shared it, and that perhaps it was just your retelling of an existing thing. I really appreciate it.

I'm generally in favor of public praise and private criticism, but this post really rubbed me the wrong way. To me it reads as a group of neurotic people getting together to try to get out of neuroticism by being even more neurotic at each other. Or, that in a quest to avoid interacting with the layer of intentions, let's go arbitrarily deep on the recursion stack at the algorithmic/strategy layer of understanding.

Also really bothered by calling a series of reactions spread over time levels of meta. Actually going meta would be paying attention to the structure of the back and forth rather than the individual steps in the back and forth.

It would help me to evaluate this post if people who have attempted this technique would report on how it went, and whether or not they found that they reached more than three levels of meta.

Very interesting analysis, thank you! This unrolling technique seems to be quite useful, at least in some situations, where, as you said people are "generally open minded enough to be willing to try out a "weird conflict resolution technique I'm experimenting with"".

One comment I have is that this is not what I understand metacognition is like, thinking about thinking about thinking ... . This appears to me more like a linear chain of inferences than a stack of them. So, in a comp-sci terms, you go through the list, rather than unroll a stack. The difference is that to get to "Bailey observes (1.1), and feels it was bad," one does not have to pop every single level from the stack, but can actually go through the chain of inferences back and forth from any element. Or at least that is what it seems like to me.

As for your point that people shy away from posting "mediocrely-explained ideas", I agree, especially if the ideas are not in the mainstream. *looks at the backlog of a dozen drafts* nope, totally not what I am doing.

See this paper arguing that humans have the ability to solve tasks that require up to seven levels of recursive metacognition, especially when those tasks are ecologically valid (i.e. the prompts are films of interactions, and the question is to pick which sentence one of the participants is more likely to say). Abstract:

Recursive mindreading is the ability to embed mental representations inside other mental representations e.g. to hold beliefs about beliefs about beliefs. An advanced ability to entertain recursively embedded mental states is consistent with evolutionary perspectives that emphasise the importance of sociality and social cognition in human evolution: high levels of recursive mindreading are argued to be involved in several distinctive human behaviours and institutions, such as communication, religion, and story-telling. However, despite a wealth of research on first-level mindreading under the term Theory of Mind, the human ability for recursive mindreading is relatively understudied, and existing research on the topic has significant methodological flaws. Here we show experimentally that human recursive mindreading abilities are far more advanced than has previously been shown. Specifically, we show that humans are able to mindread to at least seven levels of embedding, both explicitly, through linguistic description, and implicitly, through observing social interactions. However, our data suggest that mindreading may be easier when stimuli are presented implicitly rather than explicitly. We argue that advanced mindreading abilities are to be expected in an extremely social species such as our own, where the ability to reason about others' mental states is an essential, ubiquitous and adaptive component of everyday life.

[ETA: the paper was actually brought to my attention by the author of the OP]

I've thought a lot about this post in the last year, and also referenced it a few times in the broader context of talking to people about ideas around common-knowledge. I think it, together with Ben's post on common knowledge communicates the core concept quite well. 

I think the key point is that situations where A thinks that B thinks that C thinks that X don't require both A and B to be purely guessing. Instead, C might have told B they think X, and A might have observed that conversation. It feels to me that when people point at the difficulty with high levels of meta, they mean the difficulty of guessing through multiple levels. The examples in this post involve just observing what's going on, which shows how it's often easy to form such states of knowledge in practice, if you don't have to guess.

I don't particularly agree about relevance of the chunking words, as the examples are only specific models of their meaning, the meaning itself can't be unrolled into a deep explicit chain of meta, it's more of a forest of possibilities than a chain, which is unlike the usual setting of "many levels of meta". It does turn into more of a chain when there's enough data, but then the words themselves become only a small part of the relevant state of knowledge.

1.4) Alex reflects on feeling judged, doesn’t like it, and concludes that Bailey is “a downer”.

Unrolling of referents: Alex felt it was bad that Alex felt that Bailey felt that Alex leaving out the milk was bad.

There seems to be some ambiguity here, which I found very confusing on a first read-through, and which I’m still not sure I’m interpreting in a way that makes any sense.

Do you mean:

“Alex felt bad, due to the fact that Alex felt that Bailey felt …”

Or:

“Alex felt it was bad that Alex felt that Bailey felt …”

In other words, is it that Alex dislikes being made to feel judged? Or, is it that Alex views his own response (feeling judged) to be somehow un-virtuous?

Edit: Also, how/why does Alex conclude that Bailey is “a downer”? How does this follow from the rest of it? I don’t get this part.

Edit2: I also don’t get this part:

2.5) Charlie observes Alex treating Bailey like “a downer”, thinks this is baseless, and feels Alex is “a snob”.

A snob? What? I don’t understand how anything like the usual meanings of the concept of “being a snob” can make this reaction make sense.

Something strange is going on with your word/concept usage here, and I’m more and more perplexed about what it might be.

In extrapolating the writing time on this post, the votes on this post (53) in a few hours since writing and the method itself...

I suspect you have communication abilities that are above the average human and there's more to this method than you are putting into words in the first instance.

For example, hearing the "yes" when someone said "no" to feeling judged.

That's great for you, but unfortunate for anyone replicating. Do you have any comments on this problem?

Thank you so much for writing this! I remember reading a tumblr post that explained the main point a while back and could never find it again -because tumblr is an unsearchable memory hole- and kept needing to link it to people who got stuck on taking Eliezer's joking one-liner seriously.

If you like this post but want more examples, Knots by R.D Laing is a book full of them. http://www.oikos.org/knotsen1.htm

Knots by R.D Laing is full of really on the point examples of multi-step inferences that often get condensed into a single feeling. If this post interests you at all, I think reading said book will be useful.

I think this post crystallized a concept I had not previously had and will be useful for writing about rationality

This post (alongside Duncan's post Common Knowledge and Miasma) has been in my head since I read it. It has directed my attention to the many instances in which I can unroll a "simple" seeming judgment into multiple levels of modeling people's metas.

Things I'd like to see:

• This post doesn't really argue much that this happens commonly. It just gives you the idea of the mechanisms, and upon reading I went "Oh shit, of course, I see this all the time." It could be useful to see what research there is on people make these sorts of recursive mental models.
• I think to really hit the point home, a few examples should be given that are complex enough that to fully unroll them drawings are needed, and yet the simple English description still is easily parse-able ("I don't think she liked that he was being weird to everyone")

The complete unrolling of 2.5 (and thus 2.6) feel off if they are placed in the same chain of meta-reasoning. Specifically, Charlie doesn't seem like she's reacting to any chains at all, just the object-level aspect of Alex pegging Bailey as a downer. I can see how more layers of meta can arise in general, but in situations like these where a third person arrives after some events have already unfolded doesn't feel like it fits that model very well - is the claim that Charlie does a subconscious tree search for various values of X that might have caused such a chain of interactions, and then draws conclusions about the baselessness of the 'downer' brand based on that?

It seems that a large subset of issues in situations like these but perhaps more grave is that Bailey does indeed do 2.6 exactly as stated, except it's based on a non-existing chain in 2.5, leading to a quagmire of false understanding.

This post seems to be helpful, and it is nice to see that meta-level reasoning is something that happens in ordinary situations, though the exact number of levels of meta reasoning isn't particularly clear.

A point of ambiguity here seems to be when you are supposed to use this as a conflict resolution strategy, though I suspect this may be due to an imperfect understanding of this as you present it. Mostly it's just that my intuition appears to say that this isn't a fully generalized system for resolving miscommunications given willingness to go along with it.

Side note, but I really appreciated the bolded sentences marking the start and end of the 'tiring symbolic reasoning' section.

I normally give up on posts on this sort of topic precisely because I can see that I'm getting into an unknown amount of unpleasant mental effort holding all the "he said she said she said"s in my head at once. This time I could quickly gauge how much of that stuff there was, and it looked manageable, so I persevered.

I once heard an academic talking on the radio about social metacognition. He pointed out that some episode in Shakespeare's Othello involves 7 levels of metacognition. 5 of them are in the play (unfortunately I can't recall the details, but someone thinks that someone else knew that someone else deceived someone into thinking that blah blah). The sixth is the fact that the audience understands it, and the seventh is that Shakespeare knew that by writing this he could make the audience understand it. (And arguably there's an eighth, that we realize that Shakespeare knew that...)

It only took the academic a few sentences for him to summarize the whole situation and all the levels, and it was all perfectly clear. Showing how good humans are at advanced metacognition.

Also, incidentally, metacognition is crucial in professional level poker, in which multiple levels are used - e.g. by betting this I can make my opponent think that I think that he thinks that I have an ace.

From the example it looks to me like the disagreement happens immediately, but I am not sure; I may be misreading this because of the ambiguity of the word "bad" in combination with the reality-tracking of the milk spoilage and Alex's feeling judged. What I would expect to see in unrolling the referents for each part is this:

1.1) Alex leaves out the milk for 5 minutes

1.2) Bailey observes (1.1), and feels it was bad.

Unrolling of referents: Bailey felt that Alex leaving out the milk was bad.

1.3) Alex observes (1.2), and feels judged.

Unrolling of referents: Alex felt that Bailey felt that Alex is a bad person for leaving out the milk.

Bolded the difference. This might have been within the compass of the intended meaning, but the value of unrolling the referents is not clear unless it reveals a disagreement somehow, and by my reading it doesn't appear that there is any disagreement in the initial example.

When you gave your real-life example, it appeared that you noticed a likely source of disagreement, and then used the unrolling trick to get the other party to notice that the disagreement was where you already thought it was. Have any of the cases where you used this method so far been ones of mutual bafflement?

I also agree that things should be written even if the idea may not be perfect, but that imperfection is what helps improve and become a clearer model. And, Bravo on at least the first polemic to kinda support your idea about the levels of metacognition.

AND OMG does this really happen? cuz i think now i can conjecture as to why i was a social pariah. Are feelings the primary mover for most homo sapiens? I thought it might be reason but i guess this is a very specific circumstance and the parties involved do prioritize feelings and assumptions before reason? When we go around acting motivated by feelings, judgings, and assessings how others view us doesnt seem to fit the criteria for distinct layers of metacognitive states. Perhaps if one were to ask why is there judgement in the first place and in the second do i care enough to act on it? At the very least ask, 'do u think that was bad?"

I dont even understand why all these feelings are being volleyed around when they dont have to be. Are these intrinsic cognitive acts or an extrinsic ones? Is this a first person layer or a third person layer or is it the layer of the author's metacognition?

I don't think that some of your feelings (the examples' feelings) count as different levels of metacognition. The first person would still run into a self reference recursion, e.g. the feelings of their feelings and thoughts on their thoughts ad inf.. The only person for whom perhaps more levels of metacognition exists is the third person omniscient. I think that these may prove to be factors when developing ideas about levels and layers of metacognition and perhaps as an actor in this scenario, ironically peel away and simplify the layers.

I hope that this heuristic is not the norm for our kind in society; The amount of hours wasted must be staggering.