GPT-3 and concept extrapolation

I was slightly surprised to find that even fine-tuning GPT-Neo-125M  for a long time on many sequences of letters followed by spaces, followed by a colon, followed by the same sequence in reverse, was not enough to get it to pick up the pattern - probably because the positional encoding vectors make the difference between e.g. "18 tokens away" and "19 tokens away" a rather subtle difference. However, I then tried fine-tuning on a similar dataset with numbers in between (e.g. "1 W 2 O 3 R 4 D 5 S : 5 S 4 D 3 R 2 O 1 W") (or similar representation -- can't remember exactly, but something roughly like that) and it picked up the pattern right away. Data representation matters a lot!

For the similar anagram task, I found space-separating (to avoid the BPE inconsistency/nondeterminism by forcing it to encode individual letters) seemed like it helped: https://gwern.net/GPT-3-nonfiction#anagrams

For this task, I think a worthwhile followup would be to experiment with the new edit mode.

Possibly! Though it did seem to recognise that the words were spelt backwards. It must have some backwards spelt words in its training data, just not that many.

The rain in Spain stays mainly in the plain. 

This is a famous line from the play Pygmalion by George Bernard Shaw. In the play, a character named Henry Higgins is teaching a lower-class woman named Eliza Doolittle how to speak proper English. He tells her that the rain in Spain stays mainly in the plain in order to help her remember the correct pronunciation of the word "plain."

The aim of this post is not to catch out GPT-3; it's to see what concept extrapolation could look like for a language model.

It feels like a "gotcha" rebuke, but it honestly doesn't seem like it really addresses the article's point. Unless you think GPT-3 would perform better if given more time to work on it?

Humans don't have the same goal as GPT-3, though, so that doesn't seem like a fair comparison.

Remember, you only have about 1 second to think before outputting each token.

I don’t think this is true. Humans can decide to think longer on harder problems in a way GPT-3 can’t. Our “architecture” is fundamentally different from GPT-3 in that regard.

Also, our ability to think for longer fundamentally changes how we do concept extrapolation. Given a tricky extrapolation problem, you wouldn’t just spit out the first thing to enter your mind. You’d think about it.

If GPT-3 has an architectural limitation that prevents it from doing concept extrapolation in a human-like manner, we shouldn’t change our evaluation benchmarks to avoid “unfairly” penalizing GPT-3. We should acknowledge that limitation and ask how it impacts alignment prospects.