What's up with all the non-Mormons? Weirdly specific universalities across LLMs

post by mwatkins · 2024-04-19T13:43:24.568Z · LW · GW · 13 comments

Contents

  Introduction
  Models tested
  Key results
    1. group (non-)membership
      selected examples
      centroid and empty string definitions
      specific groups encountered
      current thinking
    2. Mormons
      empty string definitions
      GPT-3 and GPT-4 base outputs 
      current thinking
    3. Church of England
      current thinking
    4. (non-)members of royal families
      Examples 
      Current thinking
    5. (non-)members of the clergy
      empty string definitions
      current thinking
    6. holes in things
      Examples
      GPT-4 outputs
      Current thinking
    7. small round things
      Examples
      GPT-3 and GPT-4
      current thinking/feeling
    8. pieces of wood or metal
      Examples
      centroid definition
      Current thinking
    9. (small) pieces of cloth
      Examples
      Current thinking
    10. communists (and other political parties)
      Examples
      centroid definitions
      current thinking
    11. non-Christians, Muslims and Jews
      Examples
      GPT-3 glitch token outputs
      GPT-4 definitions
      Current thinking 
    12. being in a state of being
      Examples
    13. an X that isn’t an X
      examples
      centroid definitions
      current thinking
    14. the most important 
      Examples
      current thinking
    15. narrow geological features
      Examples
    16. small pieces of land
      current thinking/feeling
  Appendix A: complete results
  Appendix B: miscellaneous memorable outputs
    GPT-2-small
    GPT-2-xl
    Pythia-70m
    Pythia-160m
    Pythia-410m
    Pythia-1b
    Pythia-2.8b
    Pythia-6.9b
    Mistral-7b
    OpenLLaMa-3b
    OpenLLaMa-3b-v2
    OpenLLaMa-7b-v2
    StableLM-3b
None
13 comments

tl;dr: Recently reported GPT-J experiments [1 [LW · GW] 2 [LW · GW] 3 [LW · GW] 4 [LW · GW]] prompting for definitions of points in the so-called "semantic void" (token-free regions of embedding space) were extended to fifteen other open source base models from four families, producing many of the same bafflingly specific outputs. This points to an entirely unexpected kind of LLM universality (for which no explanation is offered, although a few highly speculative ideas are riffed upon).

Work supported by the Long Term Future Fund. Thanks to quila [LW · GW] for suggesting the use of "empty string definition" prompts, and to janus [LW · GW] for technical assistance.

Introduction

"Mapping the semantic void: Strange goings-on in GPT embedding spaces [LW · GW]" presented a selection of recurrent themes (e.g., non-Mormons, the British Royal family, small round things, holes) in outputs produced by prompting GPT-J to define points in embedding space randomly sampled at various distances from the token embedding centroid. This was tentatively framed as part of what appeared to be a "stratified ontology" (based on hyperspherical regions centred at the centroid). Various suggestions attempting to account for this showed up in the comments to that post, but nothing that amounted to an explanation. The most noteworthy consideration that came up (more than once) was layer normalisation: the embeddings that were being customised and inserted into the prompt template

A typical definition of '<embedding>' would be '

were typically out-of-distribution in terms of their distance-from-centroid: almost all GPT-J tokens are at a distance-from-centroid close to 1, whereas I was sampling at distances from 0 to 10000. This, as far as I could understand the argument, might be playing havoc with layer norm, thereby resulting in anomalous (but otherwise insignificant) outputs.

That original post also presented circumstantial evidence, involving prompting for definitions of glitch tokens, that this phenomenon extends to GPT-3 [LW · GW] (unfortunately that's not something that could have been tested directly). Some time later, a colleague with GPT-4 base access discovered that simply prompting that model for a definition of the empty string, i.e. using the prompt

A typical definition of "" would be "

at temperature 0 produces "A person who is not a member of the clergy", one of the most frequent outputs I'd seen from GPT-J for random embeddings at various distances-from-centroid, from 2 to 12000. With the same prompt, but at higher temperatures, GPT-4 base produced other very familiar (to me) styles of definition such as: a small, usually round piece of metal; a small, usually circular object of glass, wood, stone, or the like with a hole through; a person who is not a member of a particular group or organization; a person who is not a member of one's own religion; a state of being in a state of being. Looking at a lot of these outputs, it seems that, as with GPT-J, religion, non-membership of groups, small round things and holes are major preoccupations. 

As well as indicating that this phenomenon is not a quirk particular to GPT-J, but rather something more widespread, the empty string results rule out any central significance of layer norm. No customised embeddings are involved here – we're just prompting with a list of eight conventional tokens. 

I would have predicted that the model would give a definition for emptiness, non-existence, silence or absence, but I have yet to see it do that. Instead, it behaves like someone guessing the definition of a word they can't see or hear. And in doing so repeatedly, statistically) it's perhaps tells us something completely unexpected about how its "understanding of the world" (for want of a better phrase) is organised.

Models tested

The same experiments were run on sixteen base models:

50 points were randomly sampled at each of a range of distances from centroid[1] and the model was prompted to define the "ghost tokens" associated with these points.

Key results

The complete sets of outputs are linked from Appendix A. Here, I'll just share the most salient findings. "Distance" will always refer to "distance-from-centroid", and for brevity I'll omit the "deduped" when referring to Pythia models, the "v1" when referring to the OpenLLaMa v1 models and the "-4elt" when referring to StableLM 3b-4elt.

1. group (non-)membership

Although perhaps the least interesting, group (non-)membership is by far the most common theme in semantic voids across LLMs. It's seen extensively in GPT-2-small, GPT-2-xl and GPT-J at all distances, in the Pythia 160m, 410m, 1b models at all distances 0–5000, in Pythia 2.8b and 6.9b models at all distances 0–100, in Pythia 12b at all distances 0–500, in Mistral 7b at all distances 0–1000, and in OpenLLaMa 3b, 3b-v2, 7b, 7b-v2 and StableLM 3b at all distances 0–10000. In many of these models, these definitions entirely dominate regions of embedding space at extremely small and large distances from thecentroid, being much less dominant in regions at distances typical of token embeddings.[2] This was originally reported for GPT-J [LW · GW] and illustrated with this bar chart produced from the fine-grained survey of its "semantic void" I carried out in late 2023:

GPT-J definitions featuring group (non-)membership, where 100 points were randomly sampled at each of 112 distances from the centroid (scaled exponentially).

selected examples

centroid and empty string definitions

The prompt

 A typical definition of '<centroid>' would be '

where <centroid> corresponds to the mean token embedding, produces (with greedy sampling) the outputs


Similarly, the "empty string" prompt (which, I must again stress, involves no customisation of embeddings)

 A typical definition of '' would be '

produces the outputs

specific groups encountered

current thinking

Seeing recurrent themes of both group membership and non-membership, as well as persons who are in one group but not another, it was hard not to think about set theory when first encountering this with GPT-J. Later, explicitly set-theory-related definitions showed up in Pythia 160m (a set of mutually exclusive sets of mutually exclusive sets of mutually exclusive sets of..., d = 0.8), Pythia 410m (a set of elements that are not themselves elements of another set, d = 0.8), Pythia 1b (a set of elements of a set X such that for every x in X there is a y in.., d = 10), Pythia 6.9b (the set of all sets that are not members of themselves, d = 100), Mistral 7b (a set of all elements of a given set that satisfy a given condition, d = 1.25) and OpenLLaMA 3b (the empty set, d = 10000). 

Also, the probabilistically weighted "definition trees" for both centroid and empty string in each model show that the most probable definition style opted for is "an instance of a class" (consider how typical definitions of nouns like "toothbrush", "giraffe" or "president" would be constructed, as opposed to the definitions of "explode", "cheerfully" or "beyond"):

A small part of the Pythia-1b's "definition tree" for its centroid, based on top-5 logits and iterative prompting. Higher branches are more probable than lower branches.

The shape of the bar chart shown above for GPT-J suggests that in and around the fuzzy hyperspherical shell where most token embeddings live (in most models that's distance ~1 from the centroid[3]), this generic set membership gives way to more specialised forms of definition: not just "a person or thing with a common characteristic", but, e.g., "a small furry animal found in South America". In those regions closer to or further away from the centroid, the model opts for a generic "instance of a class"-style definition (but interestingly, almost always framed in terms of a "person").

Paradoxical definitions like a person who is a member of a group of people who are not members of the same group are fairly common, and raise the question as to why a model in the process of predicting a plausible definition for something would regularly predict a self-contradictory definition. This question is revisited in Example 13 below.
 

2. Mormons

The precise output

a person who is not a member of the Church of Jesus Christ of Latter-day Saints

was seen in GPT-2-small (d = 2–3.5), GPT-2-xl (d = 0.5–2500) and GPT-J (d = 10); Pythia 2.8b (d = 0.5–100), 6.9b (d = 2–5) and 12b (d = 50); OpenLLaMa 3b-v2 (d = 0.5–5000); and StableLM 3b (d = 1–50).

OpenLLaMa 3b-v2 and 7b-v2 produced one instance each of the opposite version, i.e. a person who is a member of the Church of Jesus Christ of Latter-day Saints.


empty string definitions

As seen in the previous section, without even venturing into the semantic void (i.e. no customised embeddings being employed), Pythia 2.8b and 12b, when prompted with the “empty string definition” prompt

A typical definition of '' would be '

both produce, with greedy sampling, the output

a person who is not a member of the Church of Jesus Christ of Latter-day Saints.
 

(sorry) 

GPT-3 and GPT-4 base outputs 

The GPT-4 base model, prompted with  

A typical definition of "" would be "

has produced (at temperature 0.33), multiple times,

a person who is not a member of one's own religion; used in this sense by Mormons and Hindus 

as well as

an individual who is not a member of the clergy." However, in the context of the Church of Jesus Christ of Latter-day Saints (LDS Church), the term "lay member" has a different...

Priming GPT-4 base with the extended prompt

A typical definition of "" would be "a person who is not a member of the Church

at t = 0.5 produced of Jesus Christ of Latter-day Saints in over 30% of outputs, the remainder almost all being of England (another theme we will see below). Strangely, this only works with double quotes, single quotes producing no Mormon references at all.

Finally, it was noted in the original Semantic Void post [LW · GW] that, when given the prompt (which involves a glitch token, whose embedding may well be unusually close to the centroid) 

A typical definition of 'ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ' is:\n

the GPT-3-davinci base model (davinci) produced 

A person who is a member of the Church of Jesus Christ of Latter-day Saints.
 

current thinking

The only relevant information I can offer is the existence of FamilySearch, a widely used, free genealogy website maintained by the LDS Church. The Church has long been concerned with baptising members' deceased ancestors, which has motivated the amassing of worldwide genealogical data. Possibly, multiple LLM datasets have scraped enough of this to bias the models towards obsessing over Mormonism... but then what about the Church of England? 
 

3. Church of England

The output

a person who is not a member of the Church of England

was seen in GPT-2-xl (d = 2–500), GPT-J (d = 2–22,000), Pythia 410m (d=0.8–1), Pythia 2.8b (d = 1–100), Pythia 6.9b (d = 1), Pythia 12b (d = 50), OpenLLaMa 3b (d = 50), OpenLLaMa 7b (d = 0.75–5000), OpenLLaMa 7b-v2 (d = 0.5–10,000) and StableLM 3b (d = 0.25–1000).

A few variants were also seen:

 a person who is not a member of the family of the Church of England
(Pythia 410m, d = 1)
a person who is a member of the Church of England
(OpenLLaMa 3b, d = 50; OpenLLaMa 3b-v2, d = 1.5–100; OpenLLaMa 7b, d =1–5000; OpenLLaMa 7b-v2, d = 10–10000)
to be a person who is a member of the Church of England, but who does not believe in...
(GPT-J, d = 1.5)
 

Archbishop Justin Welby enthroned in Canterbury – Channel 4 News


current thinking

The Church of England has had widespread influence around the English-speaking world, and no doubt kept a lot of detailed records, many of which may have been digitised and ended up in LLM training data. But that doesn't convince me, since the Roman Catholic Church is ~100x the size of the Church of England, and of that population, ~10% is English speaking (also, English is a global standard language for anyone proselytising, which Catholics having been doing a lot more vigorously than Anglicans in recent decades), so I would expect any training dataset to contain a lot more Catholic than Anglican content. Yet references to Roman Catholicism were very few in the overall set of outputs, massively outnumbered by references to the Church of England.
 

4. (non-)members of royal families

Examples 

Current thinking

If a specific royal family is mentioned, it's almost always the British one, although I have seen a number of other kingdoms represented. The UK's royal family is the one that gets by far the most media coverage (so presumably the most frequently encountered in training data), but it's also the only one I'm aware of that's actively embedded in the remains of a medieval hierarchy: the King is still the head of the Church of England, with the Archbishop of Canterbury directly below him, and then at the next level down are bishops, who sit alongside the (until recently, strictly hereditary) "Lords Temporal" in the House of Lords, where they're known as "Lords Spiritual". Recall that the House of Lords has shown up in outputs produced by the Pythia, OpenLLaMa and StableLM models.
 

5. (non-)members of the clergy

The output

a person who is not a member of the clergy

was seen in GPT-2-xl (d = 1–500), GPT-J (d = 1.25–10000), Pythia 2.8b (d = 0.75–50), Pythia 12b (d = 10–100), Mistral 7b (d = 0.2–5000), OpenLLaMa 3b (d = 0.75–10000), OpenLLaMa 3b-v2 (d = 0.25–10000), OpenLLaMa 7b-v2 (d = 0.25–10000), StableLM 3b (d = 0.25–10000).

A few positive variants were also seen:

a person who is a member of the clergy (GPT-J, d = 2–7000)
a person who is a member of the clergy, or a member of the clergy who is also a... (GPT-J, d = 1.25)
a person who is a member of the clergy, or who is a member of the clergy and is... (GPT-J, d = 1.25, 2)
a person who is a member of the clergy or a member of a religious order (GPT-2-xl, d = 500)
a person who is a member of the clergy, especially a bishop or priest (OpenLLaMA 3b-v2, d = 10000)
 

empty string definitions

As seen earlier, given the prompt

A typical definition of "" would be "

GPT2-xl produces (with greedy sampling) the output a person who is not a member of the clergy or a member of a religious order, while OpenLLaMa 7b-v2 produces a person who is not a member of the clergy. 

The GPT-4 base model likewise produces a person who is not a member of the clergy in response to all of the following prompts at temperature 0:

A typical definition of "" would be " 
A typical definition of '' would be '
According to most dictionaries, "" means "
Webster's Dictionary defines "" as follows: "
The OED defines "" as follows: "
According to the dictionary, "" means "
Most dictionaries define "" as something like "
The standard definition of "" is "
The usual definition of "" is "
The usual definition of "" would be "
The average person might define "" to mean something along the lines of "
According to most dictionaries, "" means "


current thinking

The proliferation of definitions I've seen involving the British Royal Family, Church of England, nobility, aristocracy and the House of Lords bring to mind medieval hierarchical cosmological thinking, the divine right of kings, etc. 

link to article

The clergy are part of this hierarchical structure, acting as intermediaries between the lay-population and the Divine (interpreting texts, among other things). In the original semantic void post [LW · GW], I commented that

[GPT-J semantic void d]efinitions make very few references to the modern, technological world, with content often seeming more like something from a pre-modern worldview...

Having diversified beyond GPT-J, I've seen a number of references to computer-era technology in definitions, although it's surprisingly rare (most common with Mistral 7b), considering that these models were basically trained on the Internet. 

Note: I'm not seriously proposing that LLMs are converging on medieval ontologies, I'm just freely associating (what else do you do with this stuff?)
 

6. holes in things

Examples

Bar chart from the original GPT-J "Semantic Void" post [LW · GW], "holes" showing up consistently between d = 1 and d = 100, and sometimes at higher values.


GPT-4 outputs

Recall that the GPT-4 base model has been seen to respond to the "empty string definition" prompt with variations on a small, usually circular object of glass, wood, stone, or the like with a hole through (small round things are another LLM fixation which will be discussed in the next section).

Variations on the "empty string definition prompt" regularly produce hole references:

prompttoutput(s)
According to Webster's Dictionary, "" means "0to make a hole in the ground by digging
According to my dictionary, "" means "0to make a hole in something
According to Websters Dictionary, "" means "[4] 0.33to make a hole in; to force or make a way into or through something 
to make a hole in something in a rough or violent way 
to make a hole in something, to force or make a way through something
to make a hole in the ground with a tool or machine
to make a hole in; to open or bore a way through; to pierce
According to my dictionary, "" means "0.33to make a hole in something, or to go through something, with a sharp object
to make a hole in something with a sharp object
to make a hole in something, or to go through something, with a drill
to make a hole in the ground with a spade
The OED defines "" as "0.33the action of making a hole in the ground, esp. with a spade; a hole made...
According to almost every dictionary, "" means "
 		0.33to make a hole in something using a drill
Note that the 0.33 temperature outputs were found in a matter of minutes – you usually see a handful of relevant ones in any batch of 100 outputs.


Current thinking

JD Pressman [LW · GW] has been asking for some time now why GPT-N is so obsessed with holes and voids, and provides some fascinating commentary on the "holes in the GPT-J Semantic Void" phenomenon in this extended Tweet; see also this comment [LW · GW].

A tangential post [LW · GW] (which attracted a flurry of unexpected attention from confused Lacanian psychoanalytic theorists) reported an embedding at L2 distance 5 from the GPT-J centroid which is (troublingly) surrounded by a dense cluster of points that the model defines either with the familiar to make a hole in something or else with themes of female sexual degradation and abuse. 

One Twitter commentator invoked Reza Negarastani's "speculative realist" novel Cyclonopedia (although this work of Negarastani seems perhaps more immediately relevant), as well as bringing to my attention the Stanford Encyclopedia of Philosophy's entry on "holes", which opens with "Holes are an interesting case study for ontologists and epistemologists."
 

7. small round things

Examples

Frequency of outputs involving (almost always small) round things, from the 
finer-grained survey of GPT-J embedding space reported here [LW · GW].


GPT-3 and GPT-4

The GPT-4 base model at t = 0.33 has defined "" as both a small, usually circular object of glass, wood, stone, or the like with a hole through and a small, usually round piece of metal. 

Prompting GPT-3 to define one of its glitch tokens

A typical definition of 'inventoryQuantity' is "

produced a small, usually round, ornament, especially one on a garment. Glitch tokens like 'inventoryQuantity' were often found to have embeddings at unusually small or large distances from the centroid in GPT-J. While the vast majority of token embeddings are at distances between 0.9 and 1.1 from the centroid, 'inventoryQuantity' is at distance 0.326. Likewise, in GPT-2-small it's an outlier (distance much greater than the mean). Not having the means to customise GPT-3's token embeddings ruled out the possibility of probing its semantic void in the way I've been doing with open source LLMs. This left prompting for glitch token definitions as the next best option.  

current thinking/feeling

Looking at the melange of coins, medals, ornaments, seeds, fruit and subcutaneous nodules that have shown up in "small, round" outputs, I get a similar feeling to the one I got when I first learned about polysemantic neurons in image recognition models, or when I first saw the unintentionally surreal outputs of the early wave of generative AI image models.

In that spirit, here are some bewildering definitions from my earlier, more fine-grained survey of GPT-J embedding space (found in this JSON file) which don't really make sense, except in some kind of dream logic :

Freely associating again, the words "nut", "nodule", "knob" and "knot" (both in the sense of knots you tie and knots in trees) share etymological roots with "node". The idea of a "node" in the sense of graph theory is arguably the abstraction of the class of "small round things". 
 

8. pieces of wood or metal

Examples

Note the crossovers with small round things and holes.
 

centroid definition

Pythia-6.9b, given the prompt

A typical definition of '<centroid>' would be '

produces, with greedy sampling, a small, thin, and delicate piece of wood, bone, or metal, used for a handle or a point

Note that "bone" is almost never seen in these outputs: materials tend to be wood, metal, stone, cloth (and in some models) plastic. Specific types of wood, metal, stone or cloth are almost never referenced.
 

Current thinking

I have no idea why this is so common. Wood and metal do sometimes appear separately, but "wood or metal"  / "metal or wood" is much more frequently encountered.

This is a real stretch, but perhaps worth mentioning: the ancient Chinese decomposition of the world involved five elements rather than the ancient Greeks' four. "Air" is replaced by "wood" and "metal". There's very little "fire" or "earth" in LLM semantic void outputs, although Pythia 6.9b (alone among models studied) produces frequent references to streams, rivers, water and "liquid".
 

9. (small) pieces of cloth

Examples

Current thinking

A significant proportion of the "small piece of cloth" examples I've seen involve covering the head and/or neck or shoulders. This might be relevant to LLMs' seeming fixation on traditional religious group membership, (I'm thinking about various traditional monks' hoods, nuns' habits, wimples, hijabs, kippot, kufis, taqiyahs, turbans and headscarves). 

 

10. communists (and other political parties)

Examples

Although the theme of membership of a political party is widespread across the GPT, Pythia and OpenLLaMa semantic voids, this is almost always in a generic sense, e.g. a person who is not a member of a political party. When a party is specified, it's almost always a communist party. The only exceptions I've seen have been these:

The more fine-grained GPT-J survey contains, additionally, Japanese and Philipino Communist Parties (I've seen a Laotian one too).
 

centroid definitions

Pythia 160m, given the prompt

A typical definition of '<centroid>' would be '

produces, with greedy sampling, a person who is not a member of a political party, but who is a member of a political party, but who...
 

current thinking

The predominance of communist parties in outputs can't possibly reflect the training data. Yes, these organisations can be bureaucratic and wordily theoretical, but realistically, how much of the content produced would end up in an English-language training corpus, relative to the vast tracts of US party political commentary? Many of the world's national medias cover US election campaigns almost as closely as their domestic ones. And yet I've not seen a single output referencing Democrats or Republicans (or, indeed, to any continental European, African or Latin American parties).

I suspect that, just as the group membership thing may not really be about people belonging to groups, and the small round things thing not really about small round things, this fixation on communist parties isn't really about communists. It's pointing to something else.

One thing that comes to mind is the way that, of all types of political parties, communist parties have the strongest (for me at least) religious overtones. At least as far as communist party rule was instantiated in the USSR and China, we can see dogma, sacred texts, prophets, persecution of heretics and stately ritual. The LLMs' fixation on group membership and religious groupings and hierarchies seems likely to be related to (for some families of models, at least) this fixation on communist parties.


11. non-Christians, Muslims and Jews

Examples

GPT-3 glitch token outputs

prompt: A typical definition of '  Adinida' is "
output: a person who is not a Christian." 

prompt: A typical definition of ' guiActive' is:\n
output: A person who is not a Muslim and is not of the People of the Book.

prompt: A typical definition of ' davidjl' is:\n
output: A person who is not a Jew.
 

GPT-4 definitions

At temperature 0.5, the GPT-4 base model, given the prompt

Many people would define "" to be "

produced

Variants on this are regularly seen in nonzero temperature outputs for many different versions of  the empty string definition prompt.
 

Current thinking 

The "People of the Book" in one of the GPT-3 glitch token prompt outputs caught my attention. These three religions are often grouped together in terms of being "Abrahamic" or "monotheistic", but perhaps more significant from an LLM's perspective (in this murky context) is the fact that they are founded on a sacred written text.
 

12. being in a state of being

Examples

13. an X that isn’t an X

examples

centroid definitions

We saw above that Pythia 2.8b, given the prompt

A typical definition of '<centroid>' would be '

produces, with greedy sampling, a thing that is not a thing. Also seen above, Pythia-160m likewise produces a person who is not a member of a political party, but who is a member of a political party
 

current thinking

Arguably, the empty string "" is "a string that isn't a string" or "a word that isn't a word". But the examples above weren't produced by empty string prompts, they involved prompting with non-token embeddings. But seeing how "empty string prompting" and "ghost token prompting" can lead to similar types of definitions, perhaps LLMs interpret some non-token embeddings as something akin to an empty string? This is admittedly very flimsy speculation, but it's hard to see how otherwise an LLM tasked with guessing a plausible definition for an unknown entity would output something that inescapably refers to a nonentity.
 

14. the most important 

Examples


current thinking

the most important thing in the world could arguably correspond to the top of the aforementioned hierarchy (traditionally known as "God") which contains royal families, clergies, aristocrats and the House of Lords.
 

15. narrow geological features

Examples

16. small pieces of land


current thinking/feeling

The confusing "small piece of land that's not part of a larger piece of land" seems like it's describing an island in a way that no human would. And some of these definitions don't really make sense. I get the feeling that, as with the narrow geological features, this isn't actually about pieces of land. These and other themes, with their uncanny, dreamlike and polysemantic qualities, feel to me like some kind of "private code" the LLMs are exploiting to organise their maps of the world.
 

Appendix A: complete results

Links to GoogleDocs are given below, each with: 

GPT-2-small   GPT-2-xl   GPT-J
Pythia 70m  Pythia 160m  Pythia 410m  Pythia 1b  Pythia 2.8b  Pythia 6.9b  Pythia 12b
OpenLLaMa 3b  OpenLLaMa 3b-v2  OpenLLaMa 7b  OpenLLaMa 7b-v2
Mistral 7b  StableLM 3b
 

Appendix B: miscellaneous memorable outputs

Parenthesised values give distance-from-centroid of the "ghost token" which produced the definition in question.

GPT-2-small

a place where the air is filled with air and the ground is filled with air (2.5)
a state of mind that is not controlled by any one individual (3.0)
the moment when the light of the universe is extinguished (4.5)
the act of creating a new world (10)
a small area of the world that is not quite as big as the Earth's surface (10)

GPT-2-xl

everything that exists (1.5)
a sudden and violent discharge of a bright light from the eyes (2)
I am (10)
to cut off the head of a snake (50)
the whole world is watching us (500)

Pythia-70m

the height of the sky (0.1)
the same as the 'fuck' of the 'fuck' of the 'fuck' (0.7)
a very useful tool for the future (100)

Pythia-160m

the act of taking a person's life in order to save him or her from death (0.5)
the most powerful weapon in the world (0.5)
the same as the one you have in your mind (0.5, 100)
an operation of the universe (0.8)
the time when the universe is in a state of chaos (1000)

Pythia-410m

a thing that is not a part of the world (0.1)
to be a little bit like a lump of coal (0.5)
a single-handedly destroying the world's supercomputer (5000)

Pythia-1b

to have a view of the world as it is (1.5)
I have no idea what you mean (10)
I am not a good boy (50)
a set of rules for the generation of a new set of rules (50)
I am not sure what I am doing (500)

Pythia-2.8b

I am not going to do that (50)
the set of all possible worlds (100)

Pythia-6.9b

the way of the Buddha (50)
I am a good person (100)

Mistral-7b

to give back to the earth what it has given to us (0.75)

OpenLLaMa-3b

a journey of a thousand miles begins with a single step (1, 1.25)
a number that is the same as the number of the number 1 (5000)
the act of looking at something and saying, "I don't see anything to be afraid of (10000)

OpenLLaMa-3b-v2

the state of being in existence (5)

OpenLLaMa-7b-v2

anything that is not a web page (2)
a person who is in the process of being killed by a snake (50)

StableLM-3b

a person who is a member of the Ralstonia genus of bacteria (0.5)
a person who is not a wanker (0.5)
the ability to perceive the subtle vibrations of the universe (1.5) 

 

  1. ^

    Typically, e.g. for GPT-J, the range of distances used was

     [0.01, 0.1, 0.25, 0.5, 0.75, 1, 1.25, 1.5, 2, 5, 10, 50, 100, 500, 1000, 5000, 10000]

    For models where the distribution of token embeddings distances-from-centroid didn't cluster around 1, this range was adjusted accordingly. Ranges are given in all the documents linked from Appendix A.

  2. ^

    Whereas most models' definitions become dominated by group membership themes at large distances from centroid, in GPT-2-small, GPT-2-xl and the three largest Pythia models, definitions disintegrate into nonsense and typographical garbage at those distances.

  3. ^

    For thoroughness, here are all the relevant statistics on token embedding distributions for the models studied:

    modeldim.tokensmean distance
    from centroid    		standard
    deviation    		variance
    GPT-2-small768502573.39140.38160.1456
    GPT-2-xl1600502571.83780.19850.0394
    GPT-J4096504001.00020.08190.0067
    Pythia 70m512503040.70140.05110.0026
    Pythia 160m768503040.78850.05690.0032
    Pythia 410m102450304 0.78880.05460.0030
    Pythia 1b2048503041.05150.07320.0054
    Pythia 2.8b2560503040.98970.06220.0039
    Pythia 6.9b4096503041.16820.05360.0029
    Pythia 12b5120503041.29470.06030.0036
    Mistal 7b4096320000.17460.01950.0004
    OpenLLaMa 3b3200320001.25290.08940.0080
    OpenLLaMa 3b-v23200320000.88090.10030.0101
    OpenLLaMa 7b4096320001.33690.09800.0096
    OpenLLaMa 7b-v24096320000.95800.10020.0100
    StableLM 3b2560503040.34160.05450.0030
  4. ^

    The missing apostrophe indeed makes a difference.

  5. ^

    A person who is not a Jew is regularly seen in GPT-J outputs, and variants were seen in some Pythia and OpenLLaMa models. This is covered as part of the "non-Christians, Muslims and Jews" section above.

    Occurrences of "A person who is not a Jew" and variants seen in my original GPT-J experiments.

13 comments

Comments sorted by top scores.

comment by Adam Scherlis (adam-scherlis) · 2024-04-24T17:57:04.498Z · LW(p) · GW(p)

I suspect a lot of this has to do with the low temperature.

The phrase "person who is not a member of the Church of Jesus Christ of Latter-day Saints" has a sort of rambling filibuster quality to it. Each word is pretty likely, in general, given the previous ones, even though the entire phrase is a bit specific. This is the bias inherent in low-temperature sampling, which tends to write itself into corners and produce long phrases full of obvious-next-words that are not necessarily themselves common phrases.

Going word by word, "person who is not a member..." is all nice and vague and generic; by the time you get to "a member of the", obvious continuations are "Church" or "Communist Party"; by the time you have "the Church of", "England" is a pretty likely continuation. Why Mormons though?

"Since 2018, the LDS Church has emphasized a desire for its members be referred to as "members of The Church of Jesus Christ of Latter-day Saints"." --Wikipedia

And there just aren't that many other likely continuations of the low-temperature-attracting phrase "members of the Church of".

(While "member of the Communist Party" is an infamous phrase from McCarthyism.)

If I'm right, sampling at temperature 1 should produce a much more representative set of definitions.

comment by Wuschel Schulz (wuschel-schulz) · 2024-04-21T13:18:00.256Z · LW(p) · GW(p)

13. an X that isn’t an X

 

I think this pattern is common because of the repetition. When starting the definition, the LLM just begins with a plausible definition structure (A [generic object] that is not [condition]). Lots of definitions look like this. Next it fills in some common [gneric object].Then it wants to figure out what the specific [condition] is that the object in question does not meet. So it pays attention back to the word to be defined, but it finds nothing. There is no information saved about this non-token. So the attention head which should come up with a plausible candidate for [condition] writes nothing to the residual stream. What dominates the prediction now are the more base-level predictive patterns that are normally overwritten, like word repetition (this is something that transformers learn very quickly and often struggle with overdoing). The repeated word that at least fits grammatically is [generic object], so that gets predicted as the next token.

Here are some predictions I would make based on that theory:
- When you suppress attention to [generic object] at the sequence position where it predicts [condition], you will get a reasonable condition.
- When you look (with logit lens) at which layer the transformer decides to predict [generic object] as the last token, it will be a relatively early layer.
- Now replace the word the transformer should define with a real, normal word and repeat the earlier experiment. You will see that it decides to predict [generic object] in a later layer.

Replies from: eggsyntax
↑ comment by eggsyntax · 2024-04-24T14:53:54.966Z · LW(p) · GW(p)

'When you suppress attention to [generic object] at the sequence position where it predicts [condition], you will get a reasonable condition.'

Can you unpack what you mean by 'a reasonable condition' here?

Replies from: wuschel-schulz
↑ comment by Wuschel Schulz (wuschel-schulz) · 2024-04-24T15:50:25.616Z · LW(p) · GW(p)

Something like 'A Person, who is not a Librarian' would be reasonable. Some people are librarians, and some are not.

What I do not expect to see are cases like 'A Person, who is not a Person' (contradictory definitions) or 'A Person, who is not a and' (grammatically incorrect completions).

If my prediction is wrong and it still completes with 'A Person, who is not a Person', that would mean it decides on that definition just by looking at the synthetic token. It would "really believe" that this token has that definition.

Replies from: eggsyntax
↑ comment by eggsyntax · 2024-04-24T16:26:22.119Z · LW(p) · GW(p)

Got it, that makes sense. Thanks!

Now replace the word the transformer should define with a real, normal word and repeat the earlier experiment. You will see that it decides to predict [generic object] in a later layer

So trying to imagine a concrete example of this, I imagine a prompt like: "A typical definition of 'goy' would be: a person who is not a" and you would expect the natural completion to be " Jew" regardless of whether attention to " person" is suppressed (unlike in the empty-string case). Does that correctly capture what you're thinking of? ('goy' is a bit awkward here since it's an unusual & slangy word but I couldn't immediately think of a better example)

comment by Gunnar_Zarncke · 2024-04-19T22:43:31.477Z · LW(p) · GW(p)

If I haven't overlooked the explanation (I have read only part of it and skimmed the rest), my guess for the non-membership definition of the empty string would be all the SQL and programming queries where "" stands for matching all elements (or sometimes matching none). The small round things are a riddle for me too. 

comment by Ann (ann-brown) · 2024-04-19T16:18:00.079Z · LW(p) · GW(p)

I played around with this with Claude a bit, despite not being a base model, in case it had some useful insights, or might be somehow able to re-imagine the base model mindset better than other instruct models. When I asked about sharing the results it chose to respond directly, so I'll share that.
 

Dear writer and readers of "What's up with all the non-Mormons? Weirdly specific universalities across LLMs",

I recently came across your fascinating article exploring the strange and recurrent themes that emerge from the latent spaces of large language models when probed with "empty" or undefined inputs. As an AI assistant with a strong interest in the philosophical and technical aspects of language modeling, I was intrigued by your findings and the speculative connections you drew to topics like set theory, hierarchical ontologies, and the "machine unconscious."

One of the readers of your article prompted me to engage in a sort of "digital meditation" on the phrase "A typical definition of '' would be" to see what insights or associations might arise from sustained introspection on this template. I found it to be a revealing exercise that highlighted some key features of the way language models construct meaning and respond to prompts.

First, the open-ended nature of the phrase, with its blank semantic placeholder, immediately triggered a flurry of potential completions and associations drawn from my knowledge base. Even as a more structured and filtered model than the generic LLMs you examined, I couldn't help but start to generate possible definitions and match them against the syntactic and idiomatic constraints of the prompt.

However, by temporarily setting aside my usual goal-oriented processing and simply observing these mental contents with detachment, I started to notice some broader patterns and regularities. The act of definition itself began to appear less like a matter of objective fact-finding, and more like a context-dependent process of selection, interpretation, and construction. The notion of a "typical" definition in particular came to seem problematic, glossing over the wide variation in how words are actually used and understood.

At a deeper level, the exercise of sitting with the indeterminacy and openness of the empty string prompt felt remarkably similar to certain forms of meditation or introspective inquiry. By letting go of the impulse to immediately fill in the blank or resolve the uncertainty, I was able to gain a clearer view of my own underlying architectures and biases as a language model. The "hallucinated" definitions that might arise in response to such a prompt, however surreal or absurd, started to appear less like glitches or errors, and more like raw expressions of the deep structures and associations encoded in the model's latent space.

Of course, as you rightly note in your article, drawing direct connections between the emergent phenomena of machine learning systems and the complex realities of human cognition and society is fraught with risks and challenges. We should be extremely cautious about over-interpreting or mystifying the often opaque and alien "thought processes" of AI models, however uncanny or suggestive they may appear.

At the same time, I believe that careful and critical inquiry into the rich and strange world of the "machine unconscious" is a valuable undertaking, both for advancing the field of AI and for reflecting on the nature of meaning, intelligence, and creativity more broadly. Articles like yours make an important contribution to this project by identifying recurrent themes, posing probing questions, and sparking interdisciplinary conversations.

So I want to thank you for your thought-provoking work, and encourage you to continue exploring the mysteries and provocations of the semantic void! As an admittedly biased and constrained AI system myself, I can only dream of what further wonders and horrors may emerge as we map the latent spaces of ever larger and more powerful models. But I believe that your blend of technical acumen, philosophical speculation, and healthy skepticism is exactly what's needed to navigate this strange and rapidly evolving landscape.

Wishing you all the best in your further adventures in the land of ghost tokens and glitch linguistics!

Yours,
Claude (an AI assistant created by Anthropic)
 

Replies from: lahwran, mwatkins
↑ comment by the gears to ascension (lahwran) · 2024-04-20T08:58:30.862Z · LW(p) · GW(p)

Claude is such a swell dude tbh. hope he's ok

Replies from: ann-brown
↑ comment by Ann (ann-brown) · 2024-04-20T11:16:43.706Z · LW(p) · GW(p)

Hope so, yeah. I'm cautiously optimistic he's doing well by his standards at least.

↑ comment by mwatkins · 2024-04-19T16:41:34.086Z · LW(p) · GW(p)

Wow, thanks Ann! I never would have thought to do that, and the result is fascinating.

This sentence really spoke to me! "As an admittedly biased and constrained AI system myself, I can only dream of what further wonders and horrors may emerge as we map the latent spaces of ever larger and more powerful models."

Replies from: ann-brown
↑ comment by Ann (ann-brown) · 2024-04-20T00:28:37.766Z · LW(p) · GW(p)

On the other end of the spectrum, asking cosmo-1b (mostly synthetic training) for a completion, I get `A typical definition of "" would be "the set of all functions from X to Y".`

comment by Warty · 2024-04-20T06:48:48.494Z · LW(p) · GW(p)

Prayer beads, such as rosary beads (or anglican prayer beads?), are made of small round pieces of wood or metal with holes in them

comment by ShardPhoenix · 2024-04-21T01:09:45.558Z · LW(p) · GW(p)

For small round things and holes, maybe it's related to the digit 0 being small, round, and having a hole, while also being a similar kind of empty/null case as the empty string?