Reverse-correlation: how to summon the ghost of your mental imagery

post by Malmesbury (Elmer of Malmesbury) · 2023-02-14T14:15:46.811Z · LW · GW · 0 comments

Contents

  1. Human interpretability research
  2. Concentrated extracts of stereotypes
  3. An afternoon looking at noise
  4. The random noise beauty contest
  5. Designing the ultimate food
None
No comments

Cross-posted from my blog.

As we all know, humans brains can be used to solve all kinds of problems, like classifying images, generating text or even taking important policy decisions. But these capabilities come with a downside – nobody really knows how they work. To this day human brains remain undecipherable black boxes.

This can cause trouble because humans are subject to social desirability bias – for example, if you ask a man what an attractive woman looks like, he might say that the only true beauty is kindness of heart and everyone is beautiful in their own way, but deep down he's picturing a 18 years-old with firm breasts and a 0.7 waist-to-hip ratio. He just doesn't want to sound too much like a caveman.

1. Human interpretability research

And so, human interpretability researchers – illustrious names like Jacques Lacan or Carl Jung come to mind – have spent the last few centuries developing techniques for eliciting the latent knowledge [LW · GW] of human brains. There were early attempts, such as the interpretation of dreams or the deterritorialization of the cerebro-machined desire, with mixed results.

Fortunately, there is a nerdier way to extract mental representations out of people's brains – it doesn't require tweezers or any kind of brain spoon. Instead, it requires looking at white noise for a few hours, after which a computer spits out a JPEG of what your mental images look like. Really.

Here is the basic procedure. First, take a neutral image (e.g., a neutral face) and generate thousands of noisy versions of it. This is meant to introduce subtle random distortions to the image, like darker eyes or bigger ears, changing our perception of it.

Then, show participants pairs of noisy images and ask them which one is the most [whatever concept you want to summon]. For example, Imhoff et al. 2013 asked people to classify noisy images of faces based on how likely they are to be a manager. 

After a few thousand images, take all the noise layers that you added to the images, split them into two decks based on the participants' responses, and sum them up. This combines all the alterations that make a face look manager-y, thus summoning the avatar of the quintessential manager:

They did the same thing for a nursery teacher, who everybody knows looks like this:

2. Concentrated extracts of stereotypes

This method is known as reverse-correlation. Why, I don't know, it is neither reversed nor a correlation. The main advantage for research is that it can alleviate social desirability bias, as it's hard to see what exactly you are voting for in a noisy picture. People might not be willing to admit that their mental image of a welfare recipient is an amorphous, lazy-looking ethnic minority slob. And yet, here's the output of reverse-correlation for a welfare recipient:

When Dotsch et al. (a Dutch team) asked people to classify noisy pictures based on apparent ethnicity, this is what they got:

As you’ll notice, these are the stereotype of a stereotype. It’s not a big surprise, as we are not asking about a realistic depiction of the average Moroccan, but about a concentrate of the features that make someone look Moroccan.

Also, the Moroccan man looks a bit upset. This is where the study becomes really interesting. The authors also gave the participants a questionnaire to measure their level of prejudice against Moroccan immigrants. If you put them in three categories of prejudice and make a separate ghost-image for each group, you obtain this:

For high-prejudice participants, the quintessential Moroccan looks like he’s having a really bad day. My interpretation would be that people with high prejudice are honestly scared of Moroccan immigrants (as opposed to, say, seeing them as lazy like above). To be honest, if my mental image of the average Moroccan was the guy on the left, I’d be scared too.

At this point, you can just browse the RC literature and look for funny shit. There is plenty. Here's an apologetic face versus a non-apologetic one:

Another cool thing you can do is ask people to pick pictures they think look like themselves. This is what Moon et al. 2020 did, also measuring various traits like extraversion, self-esteem, and social anxiety. This makes it possible to reconstruct how people with these traits see themselves internally:

If you don't have social anxiety, now you have an idea of what it feels like.

3. An afternoon looking at noise

If that sounds fun, it's because you haven't taken part in a RC study yourself. It takes a lot of examples before a motif starts to emerge. Think about it: encoding a 100×100 pixels image with eight different levels of gray takes 30,000 bits of information. When your participant classifies an image between two categories, you get one bit of information each time. And so, the early studies would typically ask unfortunate psych undergrads to classify as many as 20,000 noisy faces.

Since then, people have used better noise patterns (usually Gabor noise) that reduce the number of trials to about a thousand. And this opens the door to new possibilities for self-experimentation. The usual problem with conducting n=1 experiments on yourself is that you know what hypothesis is being tested and can basically choose the results. This doesn't really apply here, as you only discover the summoned image at the end (and it doesn't require any memory-erasing drug [LW · GW]!).

So I gave it a try.

4. The random noise beauty contest

What I did is classify faces based on attractiveness. There are a few reasons why I would lie to myself about what facial features I find hot. Do I like make-up? What age do I prefer? Am I still not over my ex? Is it all like evo-psych predicts? Am I trapped in the closet? Is the über-ich forcing me to suppress my Œdipus complex? I am open to any surprises.

So I bit the bullet and generated 1000 noisy versions of a neutral face, using this R package. I then classified them as hot or not. This took a bit more than an hour, divided in 4 seshs of 250 reps^[1].

Despite the serious tetris effect, going through the images was an interesting experience. It really takes very little noise to trick your brain into hallucinating an entire person. All kinds of people I know in real life popped up through the experiment. I still felt bad for the faces I rated as ugly, even though none of them are real people. On the other hand, I found it relatively easy to know what features you are voting for, so if you really want to avoid a certain outcome, you probably can.

And so, I could generate the image of a face I would find maximally attractive. Behold, my soulmate:

...

 

...

 

...

 

...

 

...

 

Ok, just kidding. Here is the real one:

That is nothing awkward like a child, a family member or, god forbid, Eliezer Yudkowski. It does, however, look like Evanescence's singer Amy Lee. It's as if this method could read into my eyes, like open doors. There are a few strange things, like the disappointed expression, the lines under the cheeks or the triangular nose. The reconstructed face's eyes are considerably lighter than the model's, in agreement with the sexual imprint theory of eye colour (my mother has blue eyes). 

What surprises me the most is the amount of make-up – usually, I would say that women look better without it. So perhaps that was just social desirability bias on my side, and make-up is actually effective on me. The alternative hypothesis is that these features are attractive to men in general, and it is the reason why women wear make-up the way they do. If any Tsimané hunter-gatherer reads this, please replicate the experiment on yourself to see if it holds cross-culturally.

5. Designing the ultimate food

The vast majority of these studies are done on faces, but I don't see why it wouldn't work on other kinds of stimuli, like music or tactile sensations. I found one precedent: Ponsot et al. 2017 had participants listen to thousands of manipulated recordings of someone saying "hello" in French and reconstructed the ultimate dominant-sounding way to greet people^[2]. Listen to it here and try it on your friends so they know who's the real alpha.

Obviously, the perfect application of the reverse-correlation technique would be to design the ultimate food. Just add random mixes of ingredients to a neutral, tasteless dish (i.e., tofu), taste it and repeat every day. After a lifetime of eating disgusting random mixes, you will be able to take the sum of all the good spice combinations and immanentize the Holy Grail of deliciousness, transcending all cultural biases and traditions. 

And then, you can die in peace, knowing that pizza with pineapple was the stairway to Nirvana all along.

1. ^{^}

   Thanks Evgeny for teaching me the vocabulary.

2. ^{^}

   As I told you, virgins and chads.

0 comments

Comments sorted by top scores.