ChatGPT understands, but largely does not generate Spanglish (and other code-mixed) text

post by Milan W (weibac) · 2022-12-23T17:40:59.862Z · LW · GW · 4 comments

Contents

  Introduction
  Understanding code-mixed text
    Understanding Spanglish (English + Spanish)
    Understanding Franglais (English + French)
    Understanding Frenspanglish (English + Spanish + French)
  Failing to generate code-mixed text
    Failing to generate Spanglish (English + Spanish)
    Failing to generate Franglais (English + French)
    Failing to generate Frenspanglish (English + Spanish + French)
  Observations
  Three tame hypotheses
  A speculative model
  Appendix 1: additional basic examples of generative failure 
  Appendix 2: Trying really hard to get ChatGPT to produce Spanglish and (almost) entirely failing
  Notes
None
4 comments

Content format: Commented chat screenshots, and then some thoughts on their implications.

Epistemic status: Exploratory

Introduction

Code-mixing is the ad-hoc mixing of two or more linguistic varieties (such as languages or dialects) in the same communicative instance. An example of code-mixing would be a sentence written with words both in Spanish and in English, or with novel words made by combining Spanish and English roots, suffixes or prefixes.

Here, I document a series of experiments designed to test ChatGPT's capabilities for understanding and generating code-mixed text. I tested it with:

The first two are well known phenomena in multilingual communities such as those in Quebec and the southwestern US, while the third is quite obscure and as far as I know does not occur naturally on a large scale. Franglais is usually regarded as the insertion of English features into French, while Spanglish as a more symmetrical phenomenon. 

This is why I decided to prompt for Franglais understanding in French, and for Spanglish and Frenspanglish understanding in English. When prompting ChatGPT to translate into a code-mixed language, I prompt in the same language the text to be translated is given.

I find that ChatGPT exhibits impressive abilities to understand text written in such code-mixes. However, despite repeated attempts at prompt engineering, I have not been able to make ChatGPT generate proper code-mixed text.

 

Understanding code-mixed text

Understanding Spanglish (English + Spanish)

ChatGPT understands Spanglish, and can translate from it into English.

Understanding Franglais (English + French)

ChatGPT understands Franglais, and can translate from it into French.

Understanding Frenspanglish (English + Spanish + French)

This took a bit of prompt engineering.

ChatGPT can understand the meaning of this mixed expression with two apostrophic contractions. However, it wrongfully explains that it is a mixture of only French and English, even replacing the French-Spanish fragment "J'puedo" with the French-only "J'peux".

 

Failing to generate code-mixed text

Note: All of these experiments were performed on a chat instance where ChatGPT had already received an explanation of the relevant code-mix.

Failing to generate Spanglish (English + Spanish)

Asked to translate an English sentence into Spanglish, ChatGPT translates into full Spanish. When the languages are reversed, the same thing happens. Example in appendix 1.

Failing to generate Franglais (English + French)

# Asked to translate an English sentence into Franglais, ChatGPT translates into full French.  When the languages are reversed, the same thing happens. Example in appendix 1.

Failing to generate Frenspanglish (English + Spanish + French)

Asked to translate back into Frenspanglish the very same message it had already translated from Frenspanglish into English, ChatGPT translates it into Spanish, and claims it has performed the request successfully. It then explains its translation piece by piece, from time to time claiming that fragments written fully in Spanish were in fact code-mixed.

 

Observations

Three tame hypotheses

A speculative model

Writing can be modeled as the iterative process of constructing a sequence of words one by one. When writing, there are two main sources contributing to your decision of which word to choose as the next one to write.

This is pretty much the distinction between type 1 (fast, intuitive) and type 2 (slow, deliberate) thinking.

ChatGPT's base model was trained to predict the next token of text in a sequence. This is analogous to the type 1 writing method in humans I just outlined. Anecdotally, it seems like people who have read more during their lives are better at it. Likewise, ChatGPT has been trained on an immense quantity of text, and is superhuman at next-token prediction [AF · GW]. Code-mixed text, however, is a rare occurrence, and as such there is insufficient data for either humans or language models to be able to generate it using only type 1 processes. Humans can get around this problem by using type 2 reasoning[1]. Language models, however, are not capable of type 2 reasoning (or an analogous artificial process), and as such can't generate code-mixed text.

 

Appendix 1: additional basic examples of generative failure 

Asked to translate a Spanish sentence into Spanglish, ChatGPT translates into full English.
Asked to translate a French sentence into Franglais, ChatGPT translates into full English.

Appendix 2: Trying really hard to get ChatGPT to produce Spanglish and (almost) entirely failing

I hazard the guess that Spanglish is the most represented code-mixed language in ChatGPT's training corpus, so I decided to try to focus my efforts here when trying to get Chat-GPT to generate a code-mixed output. All of these examples were zero-shot. That is, they were the first message in a new chat instance. There were many more attempts, here I show only the most informative ones.

Explicitly asking for only half of the words in a text in Spanish to be translated into English results in a full translation.
The same thing happens when the languages are reversed.
Inventing a situation where a fictional speaker is forced to speak Spanglish due to poor proficiency largely does not work. However, ChatGPT does sometimes manage to mix Spanish and English, albeit in a way that doesn't qualify as Spanglish. This particular example demonstrates one of the two types of mixture I have observed: [placeholder brackets] in English amid Spanish text, such as one might observe in a travel phrasebook.
This is the other mixture type I found: Repeating the same message first in Spanish and then in English.

Notes

 

Thanks to Agustín Covarrubias for feedback on an earlier version of this post, and to the anonymous OpenAI employee for an informal discussion on the matter.

 

  1. ^

    And writing code-mixed text does feel to me as slow and deliberate as an almost entirely type 2 activity.

  2. ^

    They asked me to not identify them personally in this post.

4 comments

Comments sorted by top scores.

comment by Arthur Conmy (arthur-conmy) · 2023-07-04T09:47:54.509Z · LW(p) · GW(p)

Note that this seems fairly easy for GPT-4 + a bit of nudging

Replies from: weibac
comment by Milan W (weibac) · 2023-07-06T13:57:12.243Z · LW(p) · GW(p)

Good update. Thanks.

comment by Viliam · 2022-12-23T20:35:16.038Z · LW(p) · GW(p)

Spanglish is the new captcha -- you can use it to prove that you are not a robot.

Everyone who implements this idea on their website, please make sure that only people who were already verified as not-robots can read the Spanglish texts. Otherwise you are preparing learning material for the next generation of chatbots who will be better at impersonating humans.

Or maybe this will lead to a red queen situation, where people regularly switch to new languages after the chatbots learn the old ones.

comment by Unnamed · 2022-12-24T18:02:43.875Z · LW(p) · GW(p)

I was able to get a little Spanglish with my first prompt, with "I usually" in an otherwise Spanish sentence.

Ana was telling her friend Juan about her weekends. They have an inside joke of speaking Spanglish with each other - a mix of English an Spanish. "Yo love to tomar mi coffee todos los Sabado por la morning," Ana began. She continued to talk to Juan in Spanglish, telling him about visiting her brother and her nephews. Here's what she said:

Yo love to tomar mi coffee todos los Sabado por la morning. Es mi ritual. Despues, I usually visito a mi hermano y a mis sobrinos. Los ninos son tan divertidos - siempre estan haciendo algo loco. Ayer, estuvimos en el parque y los ninos se subieron a todos los juegos. Mi sobrino mayor, Juanito, es tan competitivo. Siempre quiere ganar en todo! Pero a mi me encanta verlos jugar y divertirse. Es una de las cosas que mas disfruto de mi fin de semana.

Then I tried a couple more times with variants and got all Spanish.