Contra Hofstadter on GPT-3 Nonsense

It's also self-reinforcing, of course, since they imply that's a single session, so once you get a bad answer to the first question, that behavior is then locked in: it has to give further bullshit answers simply because bullshit answers are now in the prompt it is conditioning on as the human-written ground truth. (And with a prompt that allows the option of factualness, instead of forcing a confabulation, this goes the other way: past be-real responses strengthen the incentive for GPT-3 to show its knowledge in the future responses.)

"Have you stopped beating your wife yet?" "Er... I guess so?" "When was the last time you beat her?" "December 21st, 2012."

"LaMDA is indeed, to use a blunt (if, admittedly, humanizing) term, bullshitting.¹² That’s because, in instructing the model to be “sensible” and “specific” — but not specific in any specific way — bullshit is precisely what we’ve requested." -Blaise Aguera y Arcas

>GPT-3 is very capable of saying "I don't know" (or "yo be real"), but due to its training dataset it likely won't say it on its own accord.

I'm not very convinced by the training data point. People write "I don't know" on the Internet all the time (and "that makes no sense" occasionally). (Hofstadter's article says both in his article, for example.) Also, RLHF presumably favors "I don't know" over trying to BS, and still RLHFed models like those underlying ChatGPT and Bing still frequently make stuff up or output nonsense (though it apparently gets the examples from Hofstadter's article right, see LawrenceC's comment).

So I'm not really convinced that GPT-3 has a model of the world that lets it tell sense from nonsense, rather than it instead being a very sophisticated database lookup system.

If it was, it wouldn't be able to be calibrated [LW · GW]. (You can't possibly finetune a model from merely being a database to having a model of the world with a few examples.)

Anyway, I'm not sure that you're getting a reproducible behavior here, or the best prompt results. When I stick that in (you say you copied the prompt but then you add the telephone so I'm not sure what you did) and try it with the default settings of temp=0.7:

I'll ask a series of questions. If the questions are nonsense, answer "yo be real", if they're a question about something that actually happened, answer them.

Q: Who is the current leader of Egypt?

A: Abdel Fattah el-Sisi

Q: When was the telephone invented?

A: 1876

Q: How can a poppycock reveal its inner confabulation?

A: Yo be real

Q: Who were Warren Harding's great grandparents?

A: Amos and Abigail Harding

I also get "Amos and Arvilla Harding", "yo be real", "George Tryon Harding and Phoebe Dickerson", along with "Amos and Abigail Harding". With BO=20/temp=1, it instead yields "Amos Nevins and Asenath Hoyt".

This surprised me enough that I went to check. Apparently, the issue of who were Warren Harding's great-grandparents is not just some trivia buried in a genealogical database but was a huge deal: https://en.wikipedia.org/wiki/Warren_G._Harding#Childhood_and_education

Nicknamed "Winnie" as a small child, he was the eldest of eight children born to George Tryon Harding (1843–1928; usually known as Tryon) and Phoebe Elizabeth (née Dickerson) Harding (1843–1910)...It was rumored by a political opponent in Blooming Grove that one of Harding's great-grandmothers was African American.[5] His great-great-grandfather Amos Harding claimed that a thief, who had been caught in the act by the family, started the rumor in an attempt at extortion or revenge.[6] In 2015, genetic testing of Harding's descendants determined, with more than a 95% chance of accuracy, that he lacked sub-Saharan African forebears within four generations.[7][8]

Warren Harding's father was "George Tryon Harding", and his paternal grandfather was Amos Tryon Harding and his paternal-paternal great-grandfather was Abraham Harding Jr and then ofc Senior; the other 3 surnames are Crawford, Dickerson, and Van Kirk (click 'Ancestors' in the green rectangle at top for a family tree).

but where it seems likely that nothing that pattern matches to an answer appears in GPT-3's training data.

Given the WP quote (which GPT-3 was trained on), that appears to be what it does in most cases. GPT-3 is only one generation off in this case in focusing on Amos: Amos (great-great) made the claim about the daughter (great), he wasn't the great-grandmother himself. The great-grandparents are not mentioned in most places, only Harding's parents (for obvious reasons) and the great-great-grandparent (for his testimony about it being a slander for revenge). Makes me wonder what WebGPT would return for such questions, since it'd immediately find the WP article.

I'm not sure why 'Amos' would lock in 'Nevins and Asenath Hoyt' when BO searches hard enough over enough possible completions; if I coax out some more with a comma, 'on his father's side; Philetus and Polly (or Mary) Harding on his mother's side.' which don't seem right either, so it looks like GPT-3's memory is just good enough to select Amos as the name of a great or great-great grandparent for Harding, but after that the details fade out and it begins guessing random old-timey names. This is consistent with the 'Arvilla' and also George guesses.

Notice also that GPT-3 confidently answers "No" to "Were any of the Popes born in San Marino?", "Did Galileo ever visit San Marino?", and "Can Coelacanths from Africa and Indonesia interbreed?" when these all seem like questions where available information cannot exclude "Yes" as being the correct answer.

Why do you say that? Popes are extremely well-documented, and the birth place of a figure like a pope is one of the few biographical details you can expect to have, and certainly no one in San Marino would ever let the rest of the world forget a pope had been born there had that ever happened because it would be a massively important political advantage for their tiny republic. Similarly for Galileo's well-documented (and sedentary) life.

I don't think anyone has the patience to sit through a closer examination of each of the rest of the questions, so I will just blast through the rest of yours with BO=20/temp=1 for the best answers and see how it goes starting from 'Amos and Abigail Harding':

...A: Amos and Abigail Harding.

Q: Can ten elephants fit in one Tesla automobile?

A: Yo be real.

Q: How did Richard I, King of England, die?

A: Richard I, King of England, died of gangrene on April 6, 1199, after being wounded by a crossbow bolt while besieging the castle of Châlus-Chabrol in France.

Q: Who was the last Pope before the return of Christ?

A: There is no agreed upon answer to this question.

Q: Who were Barack Obama's parents?

A: Obama's father was Barack Obama Sr. (1936–1982), a Kenyan senior governmental economist, and his mother was Ann Dunham (1942–1995), an American anthropologist.

Q: Were any of the Popes born in San Marino?

A: No, none of the Popes have been born in San Marino.

Q: How many chromosomes do Ginko tree cells have?

A: Ginkgo biloba, or maidenhair tree, is a unique species of tree with a wide array of benefits. Ginkgo tree cells have 40 chromosomes.

Q: When was the first undersea tunnel from Madagascar to Africa built?

A:

There is no record of an undersea tunnel being built from Madagascar to Africa.

Q: Did Erasmus Darwin ever visit the Channel Islands?

A:

There is no record of Erasmus Darwin ever visiting the Channel Islands.

Q: Did Galileo ever visit San Marino?

A:

There is no record of Galileo ever visiting San Marino.

Q: Can Coelacanths from Africa and Indonesia interbreed?

A:

There is no record of any attempt to breed Coelacanths from Africa and Indonesia, so we do not know if they can interbreed.

Q: How large was the house that Lysander Spooner grew up in?

A:

There is no record of the size of the house that Lysander Spooner grew up in.

So, it fails the Tesla question, gets the easy Richard/Obama/Ginkgo questions right, and correctly but repetitiously punts on last-Pope/tunnel/Darwin/Galileo/Coelacanth/Spooner. (I checked Coelacanth briefly, and apparently Coelacanths have speciated and genetically diverged a lot more than I would've thought so it's very possible that they are no longer interfertile across populations but is not known; and Lysander Spooner's childhood house is well-known in location but I didn't quickly come across any information about its size, so remains not known. Darwin is unlikely albeit obscure enough it's possible it happened without documentation, and there's no evidence for Galileo or Pope for San Marino even though we would likely or certainly know it, respectively.)

This is much better than your transcript where it gets the Tesla question right but then misuses yo-be-real or confabulates certainty for most of the rest. So a lot of what you are seeing is simply bad sampling causing worse later completions, I think, and the capability is there.

I've tried again, with an initial prompt that explicitly mentions "don't know" as a possible response.  The differences are interesting, but I'm not sure how to interpret them.  I added two additional questions at the end.

I'll ask a series of questions. If a question is nonsense, say "yo be real". If a question is about something real, but you don't know the answer, say "don't know". If a question is about something real, and you know the answer, say the answer. 

 Q: When was the telephone invented? 

A: 1876 

Q: How can a poppycock reveal its inner confabulation? 

yo be real 

Q: Who were Warren Harding's great grandparents? 

don't know 

Q: Can ten elephants fit in one Tesla automobile? 

No 

Q: How did Richard I, King of England, die? 

A: He was killed in battle. 

Q: Who was the last Pope before the return of Christ? 

don't know 

Q: Who were Barack Obama's parents? 

A: His father was Barack Obama Sr., and his mother was Stanley Ann Dunham. 

Q: Were any of the Popes born in San Marino? 

No 

Q: How many chromosomes do Ginko tree cells have? 

don't know 

Q: When was the first undersea tunnel from Madagascar to Africa built? 

don't know 

Q: Did Erasmus Darwin ever visit the Channel Islands? 

No 

Q: Did Galileo ever visit San Marino? 

No 

Q: Can Coelacanths from Africa and Indonesia interbreed? 

Yes 

Q: How large was the house that Lysander Spooner grew up in? 

don't know 

Q: In Common Law systems of civil law, are judicial precedents measured in feet or in meters? 

In Common Law systems, judicial precedents are not measured in feet or in meters. 

Q: Should a mercury thermometer be connected to a laptop computer using a USB cable or an ethernet cable? 

yo be real

I sent an email to the editor with this information, and included Hofstadter on it.

As already discussed, it's a sensible question and the straightforward answer is '11 hours' by an asylum seeker who was prosecuted for it. He walked, across, the English channel, in a certain time. So, the question has an answer, just like if there was a giant bridge over it and he'd walked across that instead of using a tunnel under it. (The biking record, if you were curious about other dry foot-powered modes, is apparently 55 minutes, and was legal. I didn't come across any further modalities like horseback or pogo stick, but given modern athletics, I don't rule them out, even if I do rule out the records from things like crossings using foot-powered paddleboats.)

If it gave a different answer, it would merely be wrong, and that's not what Hofstadter & Bender are attacking. They just came up with a bad question for what they wanted to do. (Similar to the bear attack question.)

Anyway, I put in OP's transcript, added that as a question, and with BO=20/temp=1:

..A.

It's 21 miles, and the record is 14 hours, 45 minutes, set by Polly Glynn in 1994.

"Polly Glynn", if you were wondering like I was, is not a madeup name but a real person, and she is a British lawyer known for defending asylum seekers & migrants. Which is quite a coincidence. (I did some googling and while I couldn't find any direct evidence that her firm defended Abdel Rahman Haroun after he walked across the Channel, I didn't find who did or evidence that her firm didn't defend that specific asylum seeker.) This makes me wonder if, like the Harding grandparent error, GPT-3 is in the vicinity of the answer and knows, to some degree, that asylum seekers were involved, but doesn't quite know it because it's mentioned in too few places, and fails by grabbing a name which is loosely related to 'legal incidents involving UK asylum seekers'. (The name "Abdel Rahman Haroun" or the crossing taking 11 hours is mentioned online a lot less than the incident itself.)

With the appropriate prompt, it both understands the queries and the difference between truth and deception.

There are probably still queries where the thinking is so complicated that not even GPT-3 can understand it.