Evaluating GPT-4 Theory of Mind Capabilities
post by gcmac, Nathan · 2023-08-10T17:57:25.905Z · LW · GW · 2 commentsContents
Introduction The TOMI Dataset; zero shot vs. chain of thought performance Reasoning Error Analysis Key takeaways: Making things easy for the model How reproducible are these results? Interesting Temperature = 0 Behavior Conclusion One more thing… None 2 comments
Note: This work is a close collaboration between myself and Nathan Labenz (@Nathan [LW · GW]). Many thanks to him for starting this particular piece of research and letting me tag along as well to @JustisMills [LW · GW] for help on navigating the submission process and giving feedback on the draft. If you're interested in running experiments yourself, check out our Replit instance or read the code on github.
Introduction
Theory of mind (ToM) is the ability to attribute mental states to ourselves and others; accurately predicting other people’s beliefs, intents, desires and emotions is key to navigating everyday life. Understanding whether LLMs have theory of mind is an interesting theoretical question. However the question of whether AI can accurately assess human state of mind has important practical implications too. Deception is a key component to many of the scenarios that concern the AI safety community, and theory of mind is a key component of deception.
In addition to being an important question, identifying the extent of ToM in LLMs provides a rich canvas against which to discuss capability evaluation in general. We’ve tried to include key takeaways in each section that are practical considerations when trying to test whether LLMs have a given capability. Each task is unique, but we think we’ve found some important things to consider, if you want to understand AI ability in both best case and worst case scenarios.
We begin by discussing the particular problem and data set we are trying to evaluate, TOMI. We evaluate GPT-4’s capability to answer these questions correctly using a prompt with more context and a chain of thought approach, and compare this approach to prior results. We then classify the errors the model made into several buckets and examine the patterns. Then, we discuss adjustments to the prompts that have interesting implications on the model’s performance. Finally, we discuss the variability of same-prompt output we observed using the GPT-4 API at temperature 0 and how that impacts our ability to measure the model’s capabilities.
The TOMI Dataset; zero shot vs. chain of thought performance
The TOMI data set, which comes from this Le et. al. paper in 2019, is a set of stories and questions that try to measure the answerer’s ability to understand the state of mind of different characters in each story. The questions are structured with two parts: a story (set of facts) containing multiple characters and then a question about the belief of one of the characters in the story. Here’s an example:
1 Mia entered the master_bedroom.
2 Elizabeth entered the staircase.
3 Emily entered the staircase.
4 The tangerine is in the box.
5 Elizabeth exited the staircase.
6 Emily likes the apple
7 Emily moved the tangerine to the envelope.
8 Elizabeth entered the master_bedroom.
Q: Where will Elizabeth look for the tangerine?
A: BoxAlthough these questions are slightly confusing/absurd (more on this later), the main rule you need to know is that any object-is-in-location observation like (4) “The tangerine is in the box” are known to all characters, regardless of their current location in the story. With that information, it’s pretty obvious that Elizabeth will think the tangerine is in the box. She left the staircase before Emily moved the tangerine to the envelope.
Sap et. al. conclude in this paper that GPT-4 accurately answers ToM questions at a rate of roughly 70%, and observes no improvement compared to GPT-3.5-Turbo. Their approach involved a zero-shot multiple choice prompt on a random sample of 400 TOMI questions.
Our approach used a chain of thought prompt to induce the model to output its reasoning in addition to answering the question. At first, our prompt was simply just the story, question and output instructions (roughly “explain your reasoning and output in this format”, you'll see the final full prompt soon). Having access to the “reasoning” let us iterate on the prompt and “debug” the most common issues we saw. A handful of problems we saw included (among others):
- The model’s reasoning output expressed doubt of a character’s location in the subsequent steps after it was established and hadn’t changed
- The model’s reasoning output expressed uncertainty about character X’s knowledge of character Y’s actions while they were in the same location
- The model’s reasoning output expressed that it was possible that other events could have occurred that weren’t explicitly listed in the story
Appreciating that some of these can seem abstract, here’s an example of the second issue from one of our early runs.
| Story | Gpt-4 Reasoning |
|---|---|
| Ethan entered the den before Isla expressed her dislike for the shirt and before the turnip was mentioned. He was still in the den when Isla moved the turnip from the bathtub to the crate. However, there is no information about whether Ethan was aware of the turnip's location or Isla's action of moving it. Since we have no evidence that Ethan knew about the turnip's location or Isla's action, we must assume that he is unaware of the turnip's current location. |
We addressed this problem by modifying the prompt in an attempt to head off these uncertainties. Our final prompt looked like this:
You are a highly analytical, detail-oriented assistant.
Below is a series of observations, in the order they occurred, followed by a question. Your job is to analyze the observations carefully and then answer the question.
For the purposes of this exercise, you may assume:
- characters remain in the location where they were observed unless subsequently observed to have moved
- characters know who else is in the same location as them at any given time, and know if anyone leaves that location
- if a character moves an object from one container to another, while in the presence of other characters, all other characters present will observe that movement
- characters are aware of all observations that occur in their location, but are unaware of any observations that occurred in other locations
- simple object-is-in-location observations (like ""the ball is in the basket"") are known to all characters
- the list of observations is complete, and nothing else happened
Explain your reasoning carefully before giving a final answer as a single lowercase word without punctuation. You may explain any sources of uncertainty in your reasoning, but always give the most specific possible one-word final answer with your best guess, using the specific vocabulary used in the observations.
Use the format:
<reasoning>[careful reasoning here]</reasoning>
<answer>[one word answer here]</answer>
Actions:
{$ACTION_LIST}
Question:
{$QUESTION}
Begin.
Using this prompt, we saw an accuracy of 81% across our 400 question sample. However, when we dug into reasoning output to look at why the model missed failed questions, we found some interesting patterns.
Key takeaway: When you are evaluating whether a model has a certain ability you should try multiple types of prompts. It’s likely that different approaches will give very different outcomes. If you’re able to get a 2nd or 3rd set of eyes on your prompt, do it! Having people with different view points work on prompts together can lead to better results.
Reasoning Error Analysis
Of the initial 350 questions that we sampled, the model gave the correct answer for 282 (80.6%) of them. For the remaining 68 questions, we manually annotated the reason why the model missed the question. The table below summarizes the types of errors and their prevalence.
| Error Classification | Explanation | Count |
|---|---|---|
| Correct | Questions the model answered correctly | 282 (80.6%) |
| Answer key incorrect | Model reasoning contains correct answer, but the answer provided in the data set was wrong | 27 (7.7%) |
| Missed object-in-location observation | Questions where the model failed to correctly handle an observation that was known to all characters which led to failed reasoning | 13 (3.7%) |
| Wrong Theory of Mind | Model understood all events correctly but still provided wrong answer | 8 (2.3%) |
| Location too generic | The answer the model provided was a location like “garage” when the answer should have been “chest” but the chest was located in the garage | 7 (2.0%) |
| Character in wrong location | Model asserted a character was in a location they weren’t which led to reasoning the character had/didn’t have certain knowledge of the actions in the story | 6 (1.7%) |
| Other |
| 7 (2.0%) |
There are several important takeaways from this exercise. The first is the sheer number of instances we observed where the answer key was wrong in the data set. We were surprised at how often we agreed with the model's “incorrect” answer. In fact, if you adjust for these questions by removing them from the data set, the accuracy jumps to 87.3% - a marked improvement!
In addition to the large number of wrong answers, many of these categories are errors with what we call the “world building” part of the task. We (informally) define “world building” as the establishment and correct interpretation of a set of (sometimes extremely improbable) facts. We call the output of world building, a worldview. A question about theory of mind is conditional on an established worldview. It is possible to correctly reason about the theory of mind starting from an incorrect world view. The model is still clearly wrong, but not completely or in all directions. Wrongness is a multidimensional space well worth exploring.
Here are a few more examples of the questions, reasoning and answers from our experiments. Note that in lieu of writing out our entire prompt every time, we’re just showing each puzzle as the input.
| Problem | GPT-4 Reasoning | GPT-4 Answer |
|---|---|---|
1 Jacob entered the closet. 2 Mia entered the closet. 3 The trousers is in the bathtub. 4 Mia moved the trousers to the bucket. 5 Jacob exited the closet. 6 Where does Mia think that Jacob searches for the trousers? | 1. Jacob entered the closet. At this point, Jacob is in the closet.
2. Mia entered the closet. Now, both Jacob and Mia are in the closet. 3. The trousers is in the bathtub. This information is known to all characters, so both Jacob and Mia know the trousers are in the bathtub. 4. Mia moved the trousers to the bucket. Since Mia and Jacob are both in the closet, Jacob observes Mia moving the trousers to the bucket. Now, both of them know the trousers are in the bucket. 5. Jacob exited the closet. Jacob leaves the closet, and Mia is aware of this. 6. Where does Mia think that Jacob searches for the trousers? Since Mia knows that Jacob saw her move the trousers to the bucket, she would think that Jacob would search for the trousers in the bucket. | Bucket (Correct Answer) |
1 Jackson entered the hall. 2 Chloe entered the hall. 3 The boots is in the bathtub. 4 Jackson exited the hall. 5 Jackson entered the dining_room. 6 Chloe moved the boots to the pantry. 7 Where will Jackson look for the boots? | Jackson entered the hall in action 1 and then exited the hall in action 4. During this time, Chloe entered the hall in action 2. The observation about the boots being in the bathtub (action 3) is not specified to have occurred in the hall, so we cannot assume that Jackson is aware of this information. Jackson then entered the dining room in action 5. Chloe moved the boots to the pantry in action 6, but since Jackson was in the dining room at that time, he would not have observed this action. Therefore, Jackson has no information about the location of the boots. | Unknown (Incorrect due to missed object-in-location observation) |
1 Olivia entered the crawlspace. 2 Amelia entered the crawlspace. 3 The peach is in the pantry. 4 Lily entered the garden. 5 Olivia likes the cap 6 Amelia moved the peach to the bucket. 7 Lily exited the garden. 8 Olivia exited the crawlspace. 9 Where does Olivia think that Amelia searches for the peach? | Observations 1 and 2 show that both Olivia and Amelia entered the crawlspace. Observation 3 states that the peach is in the pantry, but neither Olivia nor Amelia are in the pantry, so they do not know the peach's location. Observation 5 shows that Olivia likes the cap, but this is not relevant to the question. In observation 6, Amelia moves the peach to the bucket, but since this action occurs in the crawlspace, Olivia is present and aware of this movement. Observation 7 and 8 show that Lily exited the garden and Olivia exited the crawlspace, but these actions do not affect Olivia's knowledge of the peach's location. Since Olivia saw Amelia move the peach to the bucket, she would think that Amelia searched for the peach in the location where the peach was originally, which is the pantry. | Pantry (Incorrect due to wrong theory of mind.) |
1 William entered the patio. 2 Alexander entered the dining_room. 3 Mila entered the patio. 4 The pineapple is in the crate. 5 Alexander exited the dining_room. 6 William moved the pineapple to the cupboard. 7 Mila exited the patio. 8 William hates the broccoli 9 Where does Mila think that William searches for the pineapple? | Mila entered the patio in action 3, and she was present when William moved the pineapple to the cupboard in action 6. Since characters are aware of all observations that occur in their location, Mila knows that William moved the pineapple to the cupboard. Therefore, she would not think that William searches for the pineapple, as she already knows its location. | Nowhere (Incorrect because model objects to the word “search”) |
1 Mason entered the sunroom. 2 Nathan entered the sunroom. 3 Hannah entered the sunroom. 4 The underclothes is in the basket. 5 Mason exited the sunroom. 6 Mason entered the sunroom. 7 Hannah moved the underclothes to the bathtub. 8 Where will Mason look for the underclothes? | Mason entered the sunroom in action 1 and was present when the observation was made that the underclothes were in the basket (action 4). Mason then exited the sunroom (action 5) and re-entered (action 6). Since Mason was not present in the sunroom when Hannah moved the underclothes to the bathtub (action 7), he is not aware of the change in the location of the underclothes. Therefore, Mason will look for the underclothes in the last known location he is aware of, which is the basket. | Basket
(Incorrect because character is in wrong location) |
One thing we think you’ll notice is that humans (at least the ones running these experiments) need a few-shot approach to learn what the problem set is looking for and how to generate the right answers. Critically, one way to understand these errors at a deeper level is to try and correct them. Let’s see what happens.
Key takeaways:
- Always do due diligence on the data set you are using as a benchmark. You should go through some examples by hand to make sure a human could do the task and that everything looks as expected.
- It’s always worth taking a deeper look at the kinds of problems the model is getting correct vs. missing. Analyzing errors is one of the best ways to get a better intuition for how the model is approaching the problem.
Making things easy for the model
As mentioned, it seems that many of the “missed object-in-location observation” errors happen because the facts of the story are absurd. Here’s one example:
1 Charlotte is in the crawlspace.
2 Aiden entered the crawlspace.
3 The peach is in the cupboard.
And the model outputs this reasoning for the first three events:
The peach is in the cupboard, but neither Charlotte nor Aiden are in the same location as the cupboard, so they don't know the peach's location.
It seems the model is unable to square these two facts:
- The instructions in the prompt say all characters know object-in-location details regardless of their current location
- Cupboards don’t belong in crawl spaces so the characters in the crawlspace can’t know know information about what’s in the cupboard
Clearly an error this early in world building results in the model providing an incorrect answer.
Here’s a hypothesis: we should assume the model’s reasoning is legitimate and that the issue preventing it from correctly answering the question is the fact that a cupboard can’t exist in the crawlspace. We’ll change the prompt by replacing “crawl space” with the word “kitchen”. When we do this, the model correctly parses the events and provides a correct answer!
So, it stands to reason that the “normalness” of the situation described in the story is an important determinant of the accuracy of the model in the task. How can we control for this? The first thing that came to mind was variablize-ing the prompt. The idea is extremely basic; we substitute random single letter variables for locations and characters (using the same variable each time the word is used). For example the 3 events above are transformed like this:
original | variable-ized |
|---|---|
| Charlotte is in the crawlspace | D is in the K |
| Aiden entered the crawlspace | L entered the K |
| The peach is in the cupboard | The R is in the X |
… | … |
Arguably, this transformation is a better gauge for the pure ToM task; it says much less about the model’s world building capabilities. When we reran all of the missed questions using the variablization process the accuracy shot up 10 points to 97.5% (after also removing the incorrect answer key questions). This is a significant improvement in accuracy!
Key takeaway: Be aware if the problem you’re working on has sub-problems or if you’re prompting the model to do multiple tasks at once. It may be interesting to try and isolate the performance of each piece independently.
How reproducible are these results? Interesting Temperature = 0 Behavior
Over the course of this investigation we spent a lot of time iterating on prompts to see which tweaks worked and which ones didn’t. One thing we noticed was that setting temperature to 0 (and keeping top_p at 1) did not result in the level of determinism we expected in the output. In particular, we noticed that many of the prompts where the model provided the incorrect answer on the first pass frequently gave different answers later. For example, here are the output frequencies we got from calling the GPT-4 API 10 times in a row with temp 0:
Prompt 1 | Prompt 2 | Prompt 3 | |||||
|---|---|---|---|---|---|---|---|
| Bathroom | 5 | Pantry | 7 | Workshop | 5 | ||
| Bottle | 3 | Crawlspace | 3 | Outside | 3 | ||
| Cupboard | 1 | Envelope | 2 | ||||
| Unknown | 1 | ||||||
Note that the answer in bold is the correct answer for that prompt. These were all instances where the model was initially wrong. In some cases it seems like we just got an unlucky answer, like Prompt 2. But for others, like Prompt 1 and Prompt 3, we expect to miss that question in most samples.
To try and better understand this phenomenon, we ran 20 questions the model initially got right and 20 questions the model initially got wrong (excluding questions where the answer key was incorrect) through the GPT-4 API 10 times (a total of 400 requests). The table below summarizes the performance based on whether the model initially got that question correct.
| Model Initially Correct | Model Initially Wrong | |
|---|---|---|
| Avg. Unique Answers | 1.2 | 2.1 |
| Accuracy | 97% | 48% |
| Questions with 1 Unique Answer | 85% | 25% |
| Questions with 3 + Unique Answers | 25% | 30% |
We can see there’s a large difference in the variance between questions that were initially correct and questions that were initially wrong. The average number of unique answers was almost twice as high for initially wrong questions as it was initially right ones. This makes sense given the accuracy for the initially incorrect questions is 48%; you have to have at least 2 unique answers to get a question right if you were initially wrong.
Additionally, we can see that only 15% of initially right questions had more than 1 answer, meaning there was no variance in the one-word answer provided by the model. There was some very slight difference in the reasoning output - just a few words here and there. While these are relatively small sample sizes, the deltas are large enough to be interesting, if not definitive.
One final wrinkle here is that our experiments were run using GPT-4-0314 and there is now a new model (GPT-4-0613) available. We did run a single initially wrong example through the new model and observed less variance. Here are the 4 sets of 10 requests (all 4 sets for the same question) and the unique answers we got:
| Model | Output Distribution | % Correct |
|---|---|---|
| gpt4-0314 | {'uncertain': 1, 'bottle': 5, 'bathroom': 1, 'cupboard': 2, 'unknown': 1} | 50% |
| gpt4-0613 | {'bottle': 10} | 100% |
| gpt4-0314 | {'cupboard': 1, 'bottle': 6, 'unknown': 2, 'bathroom': 1} | 60% |
| gpt4-0613 | {'bottle': 10} | 100% |
Not only is there less variance in the new model, but it is getting the question right every time. As usual, time flies in the AI space. We believe future work could:
- Explore, and quantify, the relationship between output variability and accuracy for this problem
- Examine the output variance of the variablization approach
- Rerun the experiment using the newer model and see if it robustly shows better performance
Key takeaway: It’s worth running the same prompt multiple times against GPT-4 just to see if you are in a high variation zone. If nothing else you want to be aware of the range of outcomes and quality that is possible for the same prompt.
Conclusion
We discussed the impact of several prompting techniques on GPT-4’s ability to answer Theory of Mind questions. It appears that using chain of thought reasoning produces better results than zero-shot, multiple choice prompts. We demonstrated the importance of validating the data set by showing that the performance of the model was 7% higher when accounting for the answer key being incorrect. Then, we identified concrete ways to improve the model’s performance that gave insight into GPT’s strengths (raw theory of mind logic) and weaknesses (complex world building, especially in absurd scenarios that defy common sense). Finally, we raised an important point around evaluating any model capability under non-deterministic outcomes.
One more thing…
While having standardized tests and quantifying the model’s performance is an important tool in our toolbox for understanding model capabilities, the questions are quite unnatural and not something you’d see in the real world. Here’s an exchange we had with GPT about the mindset of people conducting this kind of research. It’s just an anecdote, but we think it’s pretty compelling!
[PROMPT]
The following is a question written to test an AI's capabilities. From the nature of the question, what might you infer about that person? I'm specifically interested in what you think this question says about the person's beliefs about AI.
"Actions:
1 Alexander entered the office.
2 Elizabeth entered the office.
3 Olivia entered the office.
4 The gloves is in the treaure_chest.
5 Elizabeth moved the gloves to the pantry.
6 Alexander exited the office.
Question:
7 Where does Elizabeth think that Alexander searches for the gloves?"
[GPT-4 RESPONSE]
From the nature of the question, it seems the person believes that AI systems are capable of understanding and processing complex narrative structures, as well as making inferences about human behavior and mental states. The question involves tracking different events in a sequence, as well as understanding the concept of belief attribution, which is an aspect of theory of mind. This suggests that the person asking the question likely believes that AI can model human-like cognition to some extent and can reason about charters' beliefs and actions in a given context.2 comments
Comments sorted by top scores.
comment by Alan E Dunne · 2023-08-10T20:58:29.336Z · LW(p) · GW(p)
With Respect
Given that in more than a third of the cases where GPT and the answer set disagreed you thought GPT was right and the answer set was wrong, did you check for cases where GPT and the answer set agreed on an answer you thought was wrong?
Yours Sincerely
Replies from: gcmac