Syntax, semantics, and symbol grounding, simplified

post by Stuart_Armstrong · 2020-11-23T16:12:11.678Z · LW · GW · 4 comments

Contents

  Intro: the Linear A encyclopaedia
  Grounded information
  Writing the encyclopaedia
    Ungrounded information
    Syntax versus semantics
  Doing without "the world"
    Understanding the features of another agent
    Imperfect grounding: messy, empirical practice
    Language, and interpretation of the symbols of others
  GPT-n, ungrounded
None
4 comments

tl;dr: I argue that the symbol grounding problem is not an abstract philosophical problem, but a practical issue that it is useful and necessary for algorithms to solve, empirically, to at least some extent.

Intro: the Linear A encyclopaedia

Consider $E$, a detailed encyclopaedia of the world:

The inside pages are written, for some inexplicable reason, in Linear A, an untranslated Minoan script:

We also have access to $R$, a sort of Linear A Rosetta Stone, with the same texts in Linear A and in English. Just like the original Rosetta Stone, this is sufficient to translate the language.

The amount of information in $E$ is much higher than the very brief $R$. And yet $E$ on its own is pretty useless, while $E+R$ together unlock a huge amount of useful information: a full encyclopaedia of the world. What's going on here?

Grounded information

What's going on is that $E$ contains a large amount of grounded information: information about the actual world. But when the meaning of Linear A was lost that grounding was lost too: it became just a series of symbols with a certain pattern to it. Its entropy (in the information-theory sense) did not change; but now it became essentially a random sequence with that particular entropy and pattern.

Adding in $R$ restores that grounding, and so transforms $E$ back into a collection of useful information about the world; again, without changing its information content in the entropy sense.

This is the symbol grounding problem, not seen as an isosteric philosophical issue, but a practical learning problem.

Writing the encyclopaedia

To put this in symbols, let $W$ be the world, let $F_W$ be a set of features in the world, and $Q_W$ be a probability distribution over these features (see this post [? · GW] for details of the terminology).

An agent $A$ has their own feature set $F_A$, in Linear A, which corresponds to the features of the world. The correspondence is given by the map $g_A:F_W\to F_A$. The agent also knows $Q_W$.

That $A$ then sits down to write the encyclopaedia $E$; in it, they put a lot of their knowledge of $Q_W$, using the terms in $F_A$ to do so.

Thus any agent $B$ that knows $g_A$, the grounding of $A$'s symbols, can use $E$ to figure out a lot about $Q_W$ and hence about the world.

What if agent $B$ doesn't know Linear A, but does know English? Then they have their own symbol grounding $g_B:F_W\to F_B$. The encyclopaedia $E$ is almost useless to them. It still has the same amount of information, but there's no way of translating that information into $F_B$, the features that $B$ uses.

Now the role of $R$ is clear: by translating English and Linear A into each other, it defines an equivalence between $F_A$, the features of $A$, and $F_B$, the features of $B$ (specifically, it defines $g_B\circ g_A^{-1}:F_A\to F_B$).

Given this, $B$ now knows $F_A$. And, assuming they trust $A$, they can extract information on $Q_W$ from the encyclopaedia $E$. Hence they can get some of $A$'s real-world information.

Ungrounded information

Now consider $E^{\prime }$, a text with the same amount of information to $E$, but generated by a random process. There is information about in $E^{\prime }$, but its not real-world-information-about-$W$-mediated-through-$A$; it's just random information.

Maybe there could be a $R^{\prime }$ that grounds the symbols of $E^{\prime }$ in a way that makes sense for the real world. But, since $E^{\prime }$ is random, the $R^{\prime }$ would have to be huge: essentially $R^{\prime }$ would have to contain all the real world information itself.

So we can say that both $E$ and $E^{\prime }$ are ungrounded for agent $B$; but $E$ is at least potentially grounded, in that there is a short translation that will ground it. This "short translation", $R$, only works because $E$ was constructed by $A$, a grounded agent (by $g_A$), and $B$'s features are also grounded in the same world (by $g_B$). Thus all that $R$ does is translate between grounded symbols.

Syntax versus semantics

Note that $Q_W$ looks like pure syntax, not semantics. It doesn't define any features; instead it gives a (probabilistic) measure of how various features relate to each other. Thus $E$ only gives syntactic information, it seems.

But, as I noted in previous posts [LW · GW], the boundary between syntax and semantics is ambiguous.

Suppose that an agent $A$ has a feature "gavagai", another feature "snark", and let $p=Q_A(\text{snark}\mid \text{gavagai})$ be a term in their syntax, corresponding to the probability of "snark", given "gavagai".

Suppose we are starting to suspect that, semantically^[1], "gavagai" means either rabbit or cooked meal, while "snark" means fur.

Then if $p\approx 1$, "gavagai" is likely to be rabbit, since whenever gavagai is present, there is fur. Conversely, if $p\approx 0$, then "gavagai" is likely to be a meal, since gavagai means no fur around. Re-conversely, if gavagai is rabbit, $p$ is likely to be close to $1$.

None of these interpretations are definite. But the probabilities of particular syntactic and semantic interpretations go up and down in relation to each other. So purely syntactical information provides evidence for symbol grounding - because this syntax has evolved via interactions with the real world. In fact, $g_A$, $Q_A$ and even $F_A$, all evolve through interactions with the real world, and can change when new information arrives or the model features splinter^[2].

And syntax and semantics can also substitute for each other, to some extent. Suppose $A$ has only ever seen white rabbits. Then maybe "gavagai" means "white rabbit", a semantic statement. Or maybe "gavagai" means "rabbit", while $A$ also believes the syntactic statement "all rabbits are white". Either option would result in the same behaviour and pretty much the same ways of thinking, at least in circumstances $A$ has already encountered.

You could see semantics, in humans, as instinctively recognising certain features of the world, and syntax as logically putting together these features to derive other features (using rules and life experience learnt in the past). Putting it this way, it's clear why the line between syntax and semantics is vague, and can vary with time and experience^[3].

We also often only have implicit access to our own $g_A$ and our own $Q_A$, further complicating the distinction.

Doing without "the world"

In the post on model splintering [? · GW], I argued for doing without underlying "worlds", and just using the imperfect features and models of the agents.

This is needed in this setting, too. Agent $A$ might make use of the feature "human being", but that doesn't mean that the world itself needs to have a well-defined concept of "human being". It's very hard to unambiguously define "human being" in terms of physics, and any definition would be debatable. Moreover, $A$ did not use advanced physics in their own definition.

For symbol translation, though, it suffices that $B$ understands roughly what $A$ means by "human beings" - neither needs a perfect definition.

Understanding the features of another agent

So, let's change the definitions a bit. Agent $A$ has their features $F_A$ and their probability distribution over features, $Q_A$ (the syntax); the same goes for agent $B$ with $F_B$ and $Q_B$.

The grounding operators $g_A$ and $g_B$ don't map from the features $F_W$ of the world, but from the private input-output histories of the two agents, $h_A$ and $h_B$. Write $Q_A(f_A=n\mid h_A,g_A)=p$ to designate that $p$ is the probability that feature $f_A\in F_A$ is equal to $n$, given the history $h_A$ and the grounding function^[4] $g_A$.

Now, what about $B$? They will have their assessment of $A$'s internal symbols; let $f_A^{\ast B}$ be $B$'s interpretation of $A$'s symbol $f_A$.

They can compute expressions like $Q_B(f_A^{{\ast }_B}=n\mid h_B,g_B)$, the probability $B$ assigns to $A$'s feature $f_A$ being equal to $n$. It will likely reach this probability assignment by using $g_A^{\ast B}$, $Q_A^{\ast B}$ and $h_A^{\ast B}$: these are $B$'s assessment of $A$'s grounding, syntax, and history. Summing over these gives:

$$Q_B(f_A^{{\ast }_B}=n\mid h_B,g_B)=\sum _{g_A^{\ast B},Q_A^{\ast B},h_A^{\ast B}}{Q}_A^{\ast B}(f_A^{\ast b}=n\mid h_A^{\ast B},g_A^{\ast B})Q_B(g_A^{\ast B},Q_A^{\ast B},h_A^{\ast B}\mid h_B).$$

Then $B$ will consider feature $f_A$ to be perfectly grounded if there exists $f_B$, a feature in $F_B$, such that for all histories $h_B$,

$$Q_B(f_A^{{\ast }_B}=n\mid h_B,g_B)=Q_B(f_B=n\mid h_B,g_B).$$

Thus, $A$'s assessment of $f_A$ (according to $B$'s estimate) is always perfectly aligned with $B$'s assessment of $f_B$.

Imperfect grounding: messy, empirical practice

Perfect grounding is far too strong a condition, in general. It means that $B$ believes that $A$'s estimate of $f_A$ is always better than its own estimate of $f_B$.

Suppose that $f_A$ and $f_B$ referred to the number of people in a given room. Sometimes $A$ is in the room, and sometimes $B$ is.

Obviously it makes sense for $B$ to strongly trust $A$'s estimate when $A$ is in the room and $B$ is not; conversely, it would (generally) be wrong for $B$ to do so when their positions are reversed.

So sometimes $B$ will consider $A$'s estimates to be well grounded and correct[^nodistinct]; sometimes it won't. Grounding, in practice, involves modelling what the other person knows and doesn't know. We might have a some idea as to what would lead $A$ astray - eg a very convincing human-like robot in the room, combined with poor lighting.

That sounds both very complicated (if we wanted to formalise that in terms of what agent $B$ believes about what agent $A$ believes about features...) and very simple (it's exactly what human theory of mind is all about).

What this means is that, given this framework, figuring out the grounding of the symbols of another agent is a messy empirical process. It's not a question of resolving esoteric philosophical questions, but assessing the knowledge, honesty, accuracy, and so on, of another agent.

Note the empiricism here; we've moved from:

• Are the symbols of $A$ well grounded?

to:

• What are the symbols of $A$ grounded as, and in what contexts?

Language, and interpretation of the symbols of others

Those happy few who know my research well may be puzzled at this juncture. Knowing the internal features of another agent is the ultimate structured white-box model [LW · GW] - we not only know how the other agent thinks, but we assign useful labels to their internal processes. But when I introduced that concept, I argued that you could not get structured white-box models without making major assumptions. So what's going on here?

First of all, note that the definition of grounding of $f_A$ is given entirely in terms of $B$'s estimate of $A$'s features. In the limit, $B$ could be utterly wrong about everything about $A$'s internals, and still find that $A$'s symbols are grounded, even perfectly grounded.

That's because $B$ is not using $A$'s symbols at all, but their own estimate of $A$'s symbols. Maybe $A$ always feels fear and disgust when they get cold. Assume that that is the only time that $A$ feels both those sentiments at once. Then $A$ might have two symbols, "fear" and "disgust", while $B$ might model "fear-plus-disgust" as the single feature "cold". And then $B$ can use that feature, empirically and correctly, to predict the temperature. So $B$ thinks that $A$'s feature "cold" is well grounded, even if the feature doesn't even exist in $A$'s models. This is the sense in which "gavagai" meaning either "rabbit" or "undetached rabbit-part" - actually mean the same thing [LW · GW].

But for humans, there are other factors at work. Humans communicate, using words to designate useful concepts, and converge on mutually intelligible interpretations of those words, at least in common situations. A phrase like "Run, tiger!" needs to be clearly understood immediately.

Now, we humans:

1. Partially understand each other thanks to our theory of mind, and
2. Use the words we communicate with in order to define internal concepts and features.

This means that we will converge on roughly shared understandings of what words mean, at least in typical environments. This rough shared understanding explains both the untranslatability of language and why it's mainly translatable. We might not get all the linguistic nuances of the Tale of Genji, but we do know that it's about Japanese courtiers in the 11th century, and not about time-travelling robots from the future.

A last note on language: we can use it to explore concepts that we've never encountered, even concepts that don't exist. This means that, with language, we might realise that, for someone, "gavagai" means "undetached rabbit part" rather than "rabbit", because we can use linguistic concepts to imagine a distinction between those two ideas. And then we can communicate this distinction to others.

GPT-n, ungrounded

This kind of reasoning causes me to suspect that the GPT-n series of algorithms will not reach super-human levels of capability. They've achieved a lot through syntactic manipulation of texts; but their symbols are almost certainly ungrounded. Consider two hypotheses:

1. To write like a human, an agent needs a full understanding of physics, biology, and many other sciences.
2. There are simpler models that output human-like writing, with decent probability, without modelling the hard sciences.

I think there's evidence for the second hypothesis - for example, the successes of the current and past GPT-ns. It does not seem plausible that these machines are currently modelling us from electrons upwards.

But if the second hypothesis is true, then we'll expect that the GPT-ns will reach a plateau at or near the maximal current human ability. Consider two models, $M_p$ (a full physics model of humanity and enough of the universe), and $M_s$ (a simplified model of human text generation). As long as the GPT-ns are successful with $M_s$, there will be no pressure on them to develop $M_p$. Pressure can mean reinforcement learning, objective functions, or humans ranking outputs or tweaking the code. For the algorithm to converge on $M_p$, the following need to be true:

1. Using $M_p$ is significantly better than using $M_s$, so there is pressure for the algorithm to develop the better model.
2. Moving towards $M_p$, from its current model, is a sufficient improvement over $M_s$, that it will find a path towards that model.

It seems to me that 1. might be true, but 2. seems very unlikely to be true. Therefore, I don't think that GPT-ns will need or be able to ground its symbols, and hence will be restricted to human-comparable levels of ability.

We could empirically test this, in fact. Feed GPT-3 all the physics papers we have until 1904^[5]. Could GPT-3 or any of its successors generate special and general relativity from that data? I would be extremely surprised, and mildly terrified, if it did. Because it could only do so if it really understood, in a grounded way, what physics was.

Thanks to Rebecca Gorman for help with this research.

---

1. Using $g_A$ to define semantics. ↩︎

2. Like learning that "dog" and "wolf" are meaningfully different and can't be treated the same way -- or that different breeds of dogs are also different in relevant ways. In that case, the categorisation and/or modelling will shift to become more discriminatory and precise. This tends to be in areas of relevance to the human: for a dog breeder, the individual breeds of dogs are very precise and detailed categories, while everything that lives in the sea might be cheerfully dumped into the single category "fish". ↩︎

3. See how carefully learnt deductions can become instinctive, or how we can use reason to retrain our instincts. ↩︎

4. Notice that this formulation means that we don't need to distinguish the contribution of semantics ($g_A$) from that of syntax ($Q_A$): both are folded into the same expression. ↩︎

5. Maybe filter out Lorentz's papers. ↩︎

4 comments

Comments sorted by top scores.