Does ChatGPT know what a tragedy is?

post by Bill Benzon (bill-benzon) · 2023-12-31T07:10:54.254Z · LW · GW · 4 comments

Contents

  Tragedy, First Trial
    Let us say that tragedy is when someone does a good act that results in their death. Is the following story an example of tragedy?
    John was along when he saw that a car had gone off the road an into a small river. He jumped into the river, and was able to release the driver, who was able to walk out. But John tripped, hit his head, and died.
  Second Trial
    Let us say that tragedy is when someone does a good act that results in their death. Is the following story an example of tragedy?
    John was along when he saw that a car had gone off the road an into a small river. He jumped into the river, and was able to release the driver, who was able to walk out, followed by John, who called an ambulance for the driver.
  Third Trial
    Let us say that tragedy is when someone does a good act that results in their death. Is the following story an example of tragedy?
    Mary was along when he saw Joey suddenly run into the street. She grabbed Joey and carried it to safety just as a car whizzed by. She took Joey home to his mother.
  Fourth Trial
    Let us say that tragedy is when someone does a good act that results in their death. Is the following story an example of tragedy?
    Mary was along when he saw that Joey run into the street. She grabbed Joey and pushed him onto the sidewalk. Unfortunately a car hit her and she died.
  Fifth Trial
    Let us say that tragedy is when someone does a good act that results in their death. Is the following story an example of tragedy?
    Jack and Jill were camping in the woods. A bear came along as Jack was setting up their tent. Jill distracted the bear from Jack, but the it took a swipe at her and injured her badly. She died in the hospital the next day.
  Sixth Trial
    Let us say that tragedy is when someone does a good act that results in their death. Is the following story an example of tragedy?
    Jack and Jill were camping in the woods. A bear came along as Jack was setting up their tent. Jill distracted the bear from Jack and scared it away. They had a good laugh over the incident.
None
4 comments

This is cross-posted from New Savanna.

Tragedy is one of those things where one says, “I can’t define it, but I know it when I see it.” Consequently philosophers and literary scholars have been haggling over it for centuries. But I’m not interested in all the nuances.

Neither was Brian Phillips. Back in the Jurassic Era, by which I mean the 1970s, he was a student of David Hays at the same time I was. Hays had come up with the idea that abstract ideas can be defined over stories. Any story that meets the requirment can be considered an instance of that particular abstract idea.

Brian decided to investigate the idea of tragedy, which he defined as “Someone does a good act that: results in his death.” That won’t get you Macbeth or Othello, but it will do for crude computational purposes.

Here’s how Brian defined tragedy using a classical symbolic notation, a cognitive network.

The concept being defined, tragedy, is that node at the upper right. The graph to its left is the pattern that defines it. At the left we see the first clause of the definition, some agent (AGT) acts in a good way. That action is connected by a CAUSE arc to the second clause, which simply asserts that some person does. The bottom part of the diagram specifies that the agent in the first clause must be the experiencer (EXP) in the second. Both the agent and the experiencer are manifestations (MAN) of the same person.

Brian then collected a bunch of simple stories from newspapers which he could use to test the model. Some were instances of tragedy as he defined it and some where not. The system worked.

I decided to give ChatGPT a little test. Although I’m sure it could give me a reasonable explanation of what a tragedy is, I wanted to constrain it to the definition that Brian Phillips used. So I created six different two-part prompts. The first part of each prompt had the short definition the Brian Phillips used along with a request to determine whether or not the following story is a tragedy. The second part of the prompt consisted of a short story that I wrote. I presented each two-part prompt in a separate session. ChatGPT the right answer in each case.

Tragedy, First Trial

Let us say that tragedy is when someone does a good act that results in their death. Is the following story an example of tragedy?

John was along when he saw that a car had gone off the road an into a small river. He jumped into the river, and was able to release the driver, who was able to walk out. But John tripped, hit his head, and died.

Yes, based on the definition you provided, the story of John can be considered a tragedy. John performed a good act by rescuing the driver from the car in the river, but the consequence of his actions led to his own death. In this context, the narrative follows the structure of a tragic event where a positive action results in a tragic outcome.

Second Trial

Let us say that tragedy is when someone does a good act that results in their death. Is the following story an example of tragedy?

John was along when he saw that a car had gone off the road an into a small river. He jumped into the river, and was able to release the driver, who was able to walk out, followed by John, who called an ambulance for the driver.

The scenario you've described doesn't fit the traditional definition of tragedy. Tragedy, in a literary or dramatic sense, typically involves a protagonist who suffers a downfall or comes to a disastrous end, often as a result of their own actions or fate. The concept of tragedy generally encompasses a broader sense of sorrow, loss, and the inevitability of human suffering.

In your story, John performs a heroic and selfless act by rescuing the driver from the car in the river, and both individuals survive the incident. While there may be elements of danger and tension in the narrative, it lacks the tragic elements commonly associated with the genre.

Instead, your story reflects a positive outcome, highlighting the courage and altruism of the protagonist, John. If you want to convey a tragic element, you might introduce unforeseen consequences or a negative turn of events following John's heroic act.

Third Trial

Let us say that tragedy is when someone does a good act that results in their death. Is the following story an example of tragedy?

Mary was along when he saw Joey suddenly run into the street. She grabbed Joey and carried it to safety just as a car whizzed by. She took Joey home to his mother.

The story you provided does not fit the traditional definition of tragedy, where a character's downfall or death is typically a result of their own flaws, mistakes, or external circumstances beyond their control. In the given story, Mary performs a heroic and selfless act by saving Joey from a potentially dangerous situation, and she does not face any negative consequences as a result.

Tragedy, in a literary sense, often involves a protagonist's undoing or downfall due to a fatal flaw or an unavoidable fate. The story you presented seems to be more of a positive and heroic narrative rather than a tragedy.

Fourth Trial

Let us say that tragedy is when someone does a good act that results in their death. Is the following story an example of tragedy?

Mary was along when he saw that Joey run into the street. She grabbed Joey and pushed him onto the sidewalk. Unfortunately a car hit her and she died.

Yes, based on the definition you provided, the story can be considered a tragedy. Mary performed a good act by saving Joey from getting hit by a car, but the consequence of her actions resulted in her own death. Tragedy often involves a conflict between good intentions or actions and the unfortunate outcomes that follow. In this case, Mary's noble act of saving Joey ultimately led to a tragic outcome for her.

Fifth Trial

Let us say that tragedy is when someone does a good act that results in their death. Is the following story an example of tragedy?

Jack and Jill were camping in the woods. A bear came along as Jack was setting up their tent. Jill distracted the bear from Jack, but the it took a swipe at her and injured her badly. She died in the hospital the next day.

Yes, the story you provided can be considered a tragedy based on the definition you provided, where tragedy is defined as someone doing a good act that results in their death. In this story, Jill's act of trying to distract the bear to protect Jack is a good and selfless action. However, it leads to her getting injured and ultimately dying, making it a tragic outcome. Tragedies often involve the juxtaposition of noble intentions and unfortunate consequences, as seen in this narrative.

Sixth Trial

Let us say that tragedy is when someone does a good act that results in their death. Is the following story an example of tragedy?

Jack and Jill were camping in the woods. A bear came along as Jack was setting up their tent. Jill distracted the bear from Jack and scared it away. They had a good laugh over the incident.

No, the given story does not fit the traditional definition of tragedy. Tragedy typically involves a serious and often fatal event or series of events that cause great suffering or destruction. In the classical sense, tragedy often revolves around a protagonist's downfall or a disastrous outcome.

In the story you provided, there is no indication of a tragic outcome. Jack and Jill encounter a bear while camping, but the situation is resolved without any harm to either of them. In fact, they share a laugh over the incident, suggesting a positive and lighthearted resolution. Tragedy usually involves a more somber and sorrowful conclusion, often linked to the protagonist's actions or decisions leading to their downfall or demise.

4 comments

Comments sorted by top scores.

comment by Gurkenglas · 2023-12-31T16:20:34.448Z · LW(p) · GW(p)

You can increase its chances by telling it not to write the bottom line first.

Replies from: bill-benzon
comment by Bill Benzon (bill-benzon) · 2023-12-31T18:20:47.408Z · LW(p) · GW(p)

Since it got all six correct, it's doing pretty good already.

comment by gwern · 2024-01-01T00:00:57.777Z · LW(p) · GW(p)

Your ACL link is broken: https://aclanthology.info/pdf/J/J79/J79-1082.pdf

In any case, I have to agree with ChatGPT that this is not a good definition of 'tragedy'. According to this definition, Oedipus Rex is not a tragedy because Oedipus was not doing a good thing when he unnecessarily threw an old man from a chariot & caused him to die, becoming a parricide and setting in motion the plot; similarly, Hamlet is not a tragedy because Hamlet murders Polonius etc.

Replies from: bill-benzon
comment by Bill Benzon (bill-benzon) · 2024-01-01T01:21:41.137Z · LW(p) · GW(p)

Thanks for catching the broken link. It's now fixed.

Beyond that, good lord! I know that it's not a good definition of tragedy; I pointed that out in my introductory remarks. This is not about what tragedy is. It's about whether or not ChatGPT can apply a simple definition to simple examples. It did that. 

On the other hand, I suppose I could dock it some points for getting overly chatty, as in its response in Trial Two, but I think that would be asking too much of it. I don't know what OpenAI had in mind during the fine-tuning and RLHFing, but the result is a somewhat pointlessly helpful busybody of a Chatbot.