ChatGPT is settling the Chinese Room argument

I think you're misunderstanding the Chinese Room argument, which purports to show that the program being run in the room doesn't understand anything, regardless of how good its input-output behaviour is.  So noting that ChatGPT sometimes appears to lack understanding says nothing about the Chinese Room argument - it just shows that ChatGPT isn't good enough for the Chinese Room argument to be relevant.

For the record, I think the Chinese Room argument is invalid, but we aren't at the point where the debate is of practical importance.

contains some encoding of the entire (very large) training set

Not just "some" encoding. It is in fact a semantic encoding. Concepts that are semantically close together are also close together in the encoding.  It also doesn't find answers from a set. It actually translates its internal semantic concept into words, and this process is guided by flexible requirements.

You can do stuff like tell it to provide the answer in consonants only, or in the form of a song, or pirate speak, even though the training data would barely have any relevant examples. 

It is instructive to talk to Chatgpt for 10 mins to see what it knows about the rules and strategies for playing tic-tac-toe and then ask it to actually play a game of tic-tac-toe with you using those same strategies. In my experiment, I came to 2 conclusions:

1. Chatgpt knows everything there is to know about tic-tac-toe and the best strategies for playing it.
2. Chatgpt is entirely incapable of applying the knowledge it has for (1) in an actual game of tic-tac-toe it is playing with you.

Chatgpt is merely a generative language model: which means it is good at picking the next word given the prompt and the previous words it has output. Sadly, it knows little to nothing about the meaning of those words. It can't use what it knows about a topic (the rules and best strategies for playing tic-tac-toe) to benefit it in relevant scenarios (when playing an actual game of tic-tac-toe). To Chatgpt, (1) and (2) have almost nothing to do with each other because they rarely occur in the same conversation in its training data. Despite this model being able to obviously learn it has no understanding of the things its learned. It cannot use those things to learn new things. In a sense, its web of knowledge is highly disconnected.

What you are describing in your blog post has not been my experience. ChatGPT can definitely summarize my own chat it. It is also capable of using information from one domain to another.

That said, ChatGPT sometimes makes mistakes. But this is not an argument against ChatGPT being able to understand language. It only means that having access to textual information alone is not enough to reason accurately about the physical world. This is a surmountable limitation.

I just came here to clarify that ChatGPT can generalize to some extend and hence possess some sort of understanding of the natural world. Who are we to judge that his algorithms are intrinsically worse than ours and that he doesn't understand 'in the proper way' ?