A note on 'semiotic physics'

post by metasemi (Metasemi) · 2023-02-11T05:12:30.607Z · LW · GW · 13 comments

Contents

    Introduction
    TL;DR
  Trajectories
  Simulators are multiverse generators
  Semiotic physics
  The semantic realm and the physical realm
None
13 comments

Introduction

This is an attempt to explain to myself the concept of semiotic physics that appears in the original Simulators [AF · GW] post by janus [AF · GW] and in a later post [? · GW] by Jan Hendrik Kirchner [AF · GW]. Everything here comes from janus and Jan's work, but any inaccuracies or misinterpretations are all mine.

TL;DR

Trajectories

GPT, the prototypical simulator [AF · GW], is often said to "predict the next token" in a sequence of text. This is true as far as it goes, but it only partially describes typical usage, and it misses a dynamic that's essential to GPT's most impressive performances. Usually, we don't simply have GPT predict a single token to follow a given prompt; we have it roll out a continuous passage of text by predicting a token, appending that token to the prompt, predicting another token, appending that, and so on.

Thinking about the operation of the simulator within this autoregressive loop better matches typical scenarios than thinking about single token prediction, and is thus a better fit to what we typically mean when we talk about GPT. But there's more to this distinction than descriptive point of view. Crucially, the growing sequence of prompt+output text, repeatedly fed back into the loop, preserves information and therefore constitutes state, like the tape of a Turing machine.

In the Simulators [AF · GW] post, janus writes:

I think that implicit type-confusion is common in discourse about GPT. “GPT”, the neural network, the policy that was optimized, is the easier object to point to and say definite things about. But when we talk about “GPT’s” capabilities, impacts, or alignment, we’re usually actually concerned about the behaviors of an algorithm which calls GPT in an autoregressive loop repeatedly writing to some prompt-state...

The Semiotic physics [? · GW] post defines the term trajectory to mean the sequence of tokens—prompt plus generated-output-so-far—after each iteration of the autoregressive loop. In semiotic physics, as is common in both popular and technical discourse, by default we talk about GPT as a generator of (linguistic) trajectories, not context-free individual tokens.

Simulators are multiverse generators

GPT's token-by-token production of a trajectory is stochastic: at each autoregressive step, the trained model generates an output probability distribution over the token vocabulary, samples from that distribution, and appends the sampled token to the growing trajectory. (See the Semiotic physics [AF · GW] post for more detail.)

Thus, every token in the generated trajectory is a branch point in the sense that other possible paths would be followed given different rolls of the sampling dice. The simulator is a multiverse generator analogous to (both weak and strong versions of) the many-worlds interpretation of quantum mechanics.[2] janus (unpublished) says "GPT is analogous to an indeterministic time evolution operator, sampling is analogous to wavefunction collapse, and text generated by GPT is analogous to an Everett branch in an implicit multiverse."

Semiotic physics

It's in this analogical sense that a simulator like GPT implements a "physics" whose "elementary particles" are linguistic tokens.

Like real-world physics, the simulator's "physics" leads to emergent phenomena of immediate significance to human beings. In real-world physics, these emergent phenomena include stars and snails; in semiotic physics, they're the stories the simulators tell and the simulacra [AF · GW] that populate them. Insofar as these are unprecedented rhymes with human cognition, they merit investigation for their own sake. Insofar as they're potentially beneficial and/or dangerous on the alignment landscape, understanding them is critical.[3]

Texts written by GPT include dynamic representations of extremely complex, sometimes arguably intelligent entities (simulacra) in contexts such as narrations; these entities have trajectories of their own, distinct from the textual ones they supervene on; they have continuity within contexts that, though bounded, encompass hundreds or thousands of turns of the autoregressive crank; and they often reflect real-world knowledge (as well as fictions, fantasies, fever dreams, and gibberish). They interact with each other and with external human beings.[4] As janus puts it in Simulators [AF · GW]:

I have updated to think that we will live, however briefly, alongside AI that is not yet foom’d but which has inductively learned a rich enough model of the world that it can simulate time evolution of open-ended rich states, e.g. coherently propagate human behavior embedded in the real world.

As linguistically capable creatures, we experience the simulator's outputs as semantic. The tokens in the generated trajectory carry meaning, and serve as semiotic signs. This is why we refer to the simulator's physics-analogue as semiotic physics.

In real-world physics, we have formulations such as the Schrödinger equation that capture the time evolution operator of quantum mechanics in a way that allows us to consistently make reliable predictions. We didn't always have this knowledge. janus again:

The laws of physics are always fixed, but produce different distributions of outcomes when applied to different conditions. Given a sampling of trajectories – examples of situations and the outcomes that actually followed – we can try to infer a common law that generated them all. In expectation, the laws of physics are always implicated by trajectories, which (by definition) fairly sample the conditional distribution given by physics. Whatever humans know of the laws of physics governing the evolution of our world has been inferred from sampled trajectories.

With respect to models like GPT, we're analogously at the beginning of this process: patiently and directly observing [? · GW] actual generated trajectories in the hope of inferring the "forces and laws" that govern the simulator's production of meaning-laden output.[5] The Semiotic physics [? · GW] post explains this project more fully and gives numerous examples of existing and potential experimental paths.

Semiotic physics represents a naturalistic [? · GW] method of exploring the simulator from the output side that contrasts with and complements other (undoubtedly important) approaches such as "[thinking about] exactly what is in the training data", as Beth Barnes has put it [AF · GW].

The semantic realm and the physical realm

Simulators like GPT reflect a world of semantic possibilities inferred and extrapolated from human linguistic traces. Their outputs often include remarkable renditions of the real world, but the relationship between what's depicted and real-world physical law is indirect and provisional.

GPT is just as happy to simulate Harry Potter casting Expelliarmus as an engineer deploying classical mechanics to construct a suspension bridge. This is a virtue, not a flaw, of the predictive model: human discourse is indeed likely to include both types of narrations; the simulator's output distributions must do the same.

Therefore, it's a misconception to think of semiotic physics as approximating or converging on real-world physics. The relationship between the two is analogical.

Taking a cue from the original Simulators [AF(p) · GW(p)] post, which poses the question of self-supervised learning in the limit of modeling power, people sometimes ask whether the above conclusion breaks down for a sufficiently advanced simulator. At some point, this argument goes, the simulator might be able to minimize predictive loss by modeling the physical world at such a fine level of detail that humans are emulated complete with their cognitive processes. At this point, human linguistic behaviors are faithfully simulated: the simulator doesn’t need to model Harry Potter; it’s simulating the author from the physical ground up. Doesn’t this mean semiotic physics has converged to real-world physics?

The answer is no. Leaving aside the question of whether the hypothesized evolution is plausible—this is debatable [AF(p) · GW(p)]—the more important point is that even if we stipulate that it is, the conclusion still doesn’t follow, or, more precisely, doesn’t make sense. The hypothesized internalization of real-world physics would be profoundly significant, but unrelated to semiotic physics. The elementary particles and higher-level phenomena are still in disjoint universes of discourse: quarks and bosons, stars and snails (and authors) for real-world physics; tokens, stories, and simulacra for semiotic.

Well then, the inquirer may want to ask, hasn’t semiotic physics converged to triviality? It seems no longer needed or productive if an internalized physics explains everything!

The answer is no again. To see this, consider a thought experiment in which the predictive behavior of the simulator has converged to perfection based on whole-world physical modeling. You are given a huge corpus of linguistic traces and told that it was produced either by a highly advanced SSL-based simulator or by a human being; you're not told which.

In this scenario, what's your account of the language outputs produced? Is it conditional on whether the unknown source was simulator or human? In either case, the actual behaviors behind the corpus are ultimately, reductively, rooted in the laws of physics—either as internalized by the simulator model or as operational in the real world. Therefore ultimately, reductively, uselessly, the Schrödinger equation is available as an explanation. In the human case, clearly you can do better: you can take advantage of higher-level theories of semantics that have been proposed and debated for centuries. 

What then of the simulator case? Must you say that the given corpus is rooted in semantics if the source was human, but Schrödinger if it was a simulator? Part of what has been stipulated in this scenario is a predictive model that works by simulating human language behaviors, in detail, at the level of cognitive mechanism.[6] Under this assumption, the same higher-level semantic account you used for the human case is available in the simulator case too, and to be preferred over the reductive "only physics" explanation for the same reason. If your corpus was produced by micro-level simulation of human linguistic behavior, it follows that a higher-level semantics resides within the model's emulation of human cognition. In this hypothetical future, that higher-level semantic model is what semiotic physics describes. It has converged not with physics, but with human semantics.
 

  1. ^

    I recognize some may not be ready to stipulate that human-style semantics is a necessary component of the simulator's model. I think it is, but won't attempt to defend that in this brief note. Skeptics are invited to treat it as a hypothesis based on the ease and consistency with which GPT-3 can be prompted to produce text humans recognize as richly and densely meaningful, and to see testing this hypothesis as one of the goals of semiotic physics.

  2. ^

    It's in the nature of any analogy that the analogues are similar in some ways but not others. In this case, state changes in semiotic physics are many orders of magnitude coarser-grained (relative to the state) than those in quantum physics, the state space itself is infinitesimally smaller, the time evolution operator carries more information and more structure, and so on. We can look for hypotheses where things are similar and take caution where they're different, bearing in mind that the analogy itself is a prompt, not a theory.

  3. ^

    I don't attempt to explore alignment implications in this post, which is meant simply to introduce the high-level semiotic physics concept. Such issues are touched on in the original Simulators [AF · GW] post and its comments.

  4. ^

    This said, it's worth emphasizing that simulacra need not be human, or animate, or agentic at all.

  5. ^

    There's no implication or expectation that the time evolution operator of semiotic physics will be representable in such a compact form as the Schrödinger equation. The balance of information load between state and time evolution operator in the simulator is very different from the analogous balance in quantum mechanics. In the latter, a relatively simple operator transforms a vast state, while in a GPT-like system, the state is many, many, many orders of magnitude simpler, and the operator—the simulator's trained model—comparatively vast. For its dynamics to be captured in a one-line formula would imply a surprising degree of compressibility.

  6. ^

    Again, this is dubious. But it must be premised even to arrive at this scenario.

13 comments

Comments sorted by top scores.

comment by Bill Benzon (bill-benzon) · 2023-03-09T15:00:25.863Z · LW(p) · GW(p)

Have you thought of exploring the existing literature on the complex dynamics of nervous systems. It’s huge, but it does use the math you guys are borrowing from physics.

I’m thinking in particular of the work of the late Walter Freeman, who is a pioneer in the field. Toward the end of his career he began developing a concept of “cinematic consciousness.” As you know the movement in motion pictures is an illusion created by the fact the individual frames of the image are projected on the screen more rapidly than the mind can resolve them. So, while the frames are in fact still, they change so rapidly that we see motion.

First I’ll give you some quotes from Freeman’s article to give you a feel for his thinking (alas, you’ll have to read the article to see how those things connect up), then I’ll explain what that has to do with LLMs. The numbers are from Freeman’s article.

[20] EEG evidence shows that the process in the various parts occurs in discontinuous steps (Figure 2), like frames in a motion picture (Freeman, 1975; Barrie, Freeman and Lenhart, 1996).

[23] Everything that a human or an animal knows comes from the circular causality of action, preafference, perception, and up-date. It is done by successive frames of self-organized activity patterns in the sensory and limbic cortices. [...]

[35] EEG measurements show that multiple patterns self-organize independently in overlapping time frames in the several sensory and limbic cortices, coexisting with stimulus-driven activity in different areas of the neocortex, which structurally is an undivided sheet of neuropil in each hemisphere receiving the projections of sensory pathways in separated areas. [...]

[86] Science provides knowledge of relations among objects in the world, whereas technology provides tools for intervention into the relations by humans with intent to control the objects. The acausal science of understanding the self distinctively differs from the causal technology of self-control. "Circular causality" in self-organizing systems is a concept that is useful to describe interactions between microscopic neurons in assemblies and the macroscopic emergent state variable that organizes them. In this review intentional action is ascribed to the activities of the subsystems. Awareness (fleeting frames) and consciousness (continual operator) are ascribed to a hemisphere-wide order parameter constituting a global brain state. Linear causal inference is appropriate and essential for planning and interpreting human actions and personal relations, but it can be misleading when it is applied to microscopic- microscopic relations in brains.

Notice that Freeman refers to “a hemisphere-wide order parameter constituting a global brain state.” The cerebral cortex consists of 16B neurons, each with roughly 10K connections. Further, all areas of the cortex have connections with subcortical regions. That’s an awful-lot of neurons communicating in parallel in a single time step. As I recall from another article, these frames occur at a rate of 6-7 Hz.

The nervous system operates in parallel. I believe it is known that the brain exhibits a small world topology, so all neurons are within a relatively small number links from one another. Though at any moment some neurons will be more active than others, they are all active – the only inactive neuron is a dead neuron. Similarly, ANNs exhibit a high degree of parallelism. LLMs are parallel virtual machines being simulated by so-called von Neumann machines. The use of multiple cores gives a small degree of parallelism, but that’s quite small in relation to the overall number of parameters the system has.

I propose that the process of generating a single token in an LLM is comparable to a single “frame” of consciousness in Freeman’s model. All the parameters in the system are visited during a single time-step for the system. In the case of ChatGPT I believe that’s 175B parameters.

Thus the assertion that ChatGPT generates one token at a time, based on the previous string, while true, is terribly reductive and thus misleading. The appearance of a token is in fact more or less a side-effect of evolving a trajectory from the initial prompt. 

Replies from: Metasemi
comment by metasemi (Metasemi) · 2023-03-10T14:39:14.419Z · LW(p) · GW(p)

Thanks very much for these comments and pointers. I'll look at them closely and point some others at them too.

Replies from: bill-benzon, bill-benzon
comment by Bill Benzon (bill-benzon) · 2023-03-10T18:04:15.282Z · LW(p) · GW(p)

You might also look at this:

Andrew M. Saxe, James L. McClelland, and Surya Ganguli, A mathematical theory of semantic development in deep neural networks, PNAS, vol. 116, no. 23, June 4, 2019, 11537-11546, https://www.pnas.org/content/116/23/11537

Abstract: An extensive body of empirical research has revealed remarkable regularities in the acquisition, organization, deployment, and neural representation of human semantic knowledge, thereby raising a fundamental conceptual question: What are the theoretical principles governing the ability of neural networks to acquire, organize, and deploy abstract knowledge by integrating across many individual experiences? We address this question by mathematically analyzing the nonlinear dynamics of learning in deep linear networks. We find exact solutions to this learning dynamics that yield a conceptual explanation for the prevalence of many disparate phenomena in semantic cognition, including the hierarchical differentiation of concepts through rapid developmental transitions, the ubiquity of semantic illusions between such transitions, the emergence of item typicality and category coherence as factors controlling the speed of semantic processing, changing patterns of inductive projection over development, and the conservation of semantic similarity in neural representations across species. Thus, surprisingly, our simple neural model qualitatively recapitulates many diverse regularities underlying semantic development, while providing analytic insight into how the statistical structure of an environment can interact with nonlinear deep-learning dynamics to give rise to these regularities.

Replies from: bill-benzon
comment by Bill Benzon (bill-benzon) · 2023-03-21T21:40:36.703Z · LW(p) · GW(p)

BTW, the Annual Review of Condensed Matter Physics has an article on Statistical Mechanics of Deep Learning, by some people from Google Brain and Stanford. I believe the Annual Reviews are now all open access, so you might what to look around. The Annual Review of Linguistics might have some stuff for you.

comment by Bill Benzon (bill-benzon) · 2023-03-10T15:43:51.475Z · LW(p) · GW(p)

You're welcome.

comment by ejb · 2023-03-21T21:22:04.275Z · LW(p) · GW(p)

I'm familiar with semiotics and language models, but I don't understand why you're calling this "semiotic physics" instead of "computational linguistics".
 

  1. Linguistics vs. semiotics - I think you might say, that with text tokens, we're not just talking about natural language, we're talking about arbitrary symbol systems. If an LLM can program, do math, learn dyck languages, or various icons like emojis, one might say that it's working with more than just natural language. However, I'd argue that (a) this is still a vastly limited subset of sign systems, and any sign system in an LLM is being encoded into a completely symbolic (i.e. text tokens, not icons or indexes) format, and (b) "linguistics" and "language" are widely-accepted terminology for NLP and LLMs, so this seems like a barrier of communication and extra translation cost for many readers. I'd also note that even psycholinguists that study human language production also account for non-symbolic sign systems in language use such as prosody (e.g. voice loudness is or can be iconic).

    "As linguistically capable creatures, we experience the simulator's outputs as semantic. The tokens in the generated trajectory carry meaning, and serve as semiotic signs. This is why we refer to the simulator's physics-analogue as semiotic physics."  Why did we jump from "linguistics" to "sign systems"? It's not because of semantics, and none of the analyses in the simulators seminar sequence seem to rely on terms or distinctions specific to semiotics and not linguistics, e.g. symbol-icon-index, or sign-object-interpretant. 

    I'm also aware there are deep historical and theoretical connections between linguistics and semiotics, e.g. Saussure's Course of General Linguistics, but you're not mentioning any of that here. 
     
  2. Physics vs. computational - I don't understand what makes this "physics of X" instead of "computational modeling of X". I get that you're talking about learning dynamics, but there's tons of "computational" work that does just this in computational linguistics (e.g. Partha Niyogi's work on computational models of language evolution is particularly "physics-y"), cognitive science, neuroscience, and A.I., and it doesn't seem to lose anything with using "computational" instead of "physics"

    You say "In this analogical sense, a simulator such as GPT implements a "physics" whose "elementary particles" are linguistic tokens.". Linguistics is the subfield of science where the primitive units are linguistic tokens... Maybe I'm missing something here, but it seems like you can have these nice analogies with simulation and multiverses without calling it "physics".

    How does this relate to computational semiotics?
Replies from: bill-benzon, Metasemi
comment by Bill Benzon (bill-benzon) · 2023-03-23T22:29:22.914Z · LW(p) · GW(p)

If you don't mind, I'll make a remark.

If it had been up to me, which it most certainly wasn't and isn't, it would be called the complex dynamics of transformers, or perhaps LLMs, because that's what it seems to be. That's where the math is from and where it's been most developed. I just ignore the semiotics part of the name. In any event I tend to think the notion of semiotics has long been overgeneralized to the point where it has little meaning. As far as I can tell there's not much of a connection with any of the intellectual traditions that fly the semiotics flag.

As you say, there's "a barrier of communication and extra translation cost for many readers." Well, yeah, if anyone wants to publish or post this work outside of LessWrong, the terminology is likely to prove problematic. If you look around you'll find that that's an issue in several posts, communicating with the larger intellectual world. I have no idea how that's going to work out in the long run.

comment by metasemi (Metasemi) · 2023-03-23T19:31:42.143Z · LW(p) · GW(p)

Thank you for these comments - I look forward to giving the pointers in particular the attention they deserve. My immediate and perhaps naive answer/evasion is that semiotic physics alludes to a lower level analysis: more analogous to studying neural firing dynamics on the human side than linguistics. One possible response would be, "Well, that's an attempt to explain saying 'physics', but it hardly justifies 'semiotic'." But this is - in the sense of the analogy - a "physics" of particles of language in the form of embeddable tokens. (Here I have to acknowledge that the embeddings are generally termed 'semantic', not 'semiotic' - something for us to ponder.)

Replies from: ejb
comment by ejb · 2023-04-02T21:26:21.600Z · LW(p) · GW(p)

semiotic physics alludes to a lower level analysis: more analogous to studying neural firing dynamics on the human side than linguistics


Many classic debates in cognitive science and AI, e.g. between symbolism and connectionism, translate to claims about neural substrates. Most work with LLMs that I've seen abstracts over many such details, and seems in some ways more akin to linguistics, describing structure in high-level behavior, than neuroscience. It seems like there's lots of overlap between what you're talking about and Conceptual Role Semantics - here's a nice, modern treatment of it in computational cognitive science.

I think I kind of get the use of "semiotics" more than "physics". For example, with multi-modal LLMs the symbol/icon barrier begins to dissolve, so GPT-4 can reason about diagrams to some extent. The wikipedia entry for social physics provides some relevant context:

"More recently there have been a large number of social science papers that use mathematics broadly similar to that of physics, and described as "Computational social science"

comment by Bill Benzon (bill-benzon) · 2023-03-06T21:48:08.163Z · LW(p) · GW(p)

You should check out Stephen Wolfram's long post What Is ChatGPT Doing … and Why Does It Work? Scroll down to the section, Meaning Space and Semantic Laws of Motion:

We discussed above that inside ChatGPT any piece of text is effectively represented by an array of numbers that we can think of as coordinates of a point in some kind of “linguistic feature space”. So when ChatGPT continues a piece of text this corresponds to tracing out a trajectory in linguistic feature space. But now we can ask what makes this trajectory correspond to text we consider meaningful. And might there perhaps be some kind of “semantic laws of motion” that define—or at least constrain—how points in linguistic feature space can move around while preserving “meaningfulness”?

He goes on to talk visualize a trajectory as GP-2 completes a sentence. Earlier in the article – the section on Neural Nets – he talks of "attractors" and "attractor basins."

Replies from: Metasemi
comment by metasemi (Metasemi) · 2023-03-06T22:16:14.564Z · LW(p) · GW(p)

I did read this and agree with you that it's exactly the same as semiotic physics as understood here!

Replies from: bill-benzon
comment by Bill Benzon (bill-benzon) · 2023-03-07T00:48:14.892Z · LW(p) · GW(p)

Come to think of it, I have a working paper that speculates on this sort of thing: Virtual Reading: The Prospero Project Redux.

You might also look at (notice the date): E. Alvarez-Lacalle, B. Dorow, J.-P. Eckmann, and E. Moses. Hierarchical structures induce long-range dynamical correlations in written texts. PNAS Vol. 103 no. 21. May 23, 2006: 7956–7961. doi: 10.1073/pnas.0510673103 Here's the abstract:

Thoughts and ideas are multidimensional and often concurrent, yet they can be expressed surprisingly well sequentially by the translation into language. This reduction of dimensions occurs naturally but requires memory and necessitates the existence of correlations, e.g., in written text. However, correlations in word appearance decay quickly, while previous observations of long-range correlations using random walk approaches yield little insight on memory or on semantic context. Instead, we study combinations of words that a reader is exposed to within a “window of attention,” spanning about 100 words. We define a vector space of such word combinations by looking at words that co-occur within the window of attention, and analyze its structure. Singular value decomposition of the co-occurrence matrix identifies a basis whose vectors correspond to specific topics, or “concepts” that are relevant to the text. As the reader follows a text, the “vector of attention” traces out a trajectory of directions in this “concept space.” We find that memory of the direction is retained over long times, forming power-law correlations. The appearance of power laws hints at the existence of an underlying hierarchical network. Indeed, imposing a hierarchy similar to that defined by volumes, chapters, paragraphs, etc. succeeds in creating correlations in a surrogate random text that are identical to those of the original text. We conclude that hierarchical structures in text serve to create long-range correlations, and use the reader’s memory in reenacting some of the multidimensionality of the thoughts being expressed.