Favorite colors of some LLMs.

post by weightt an (weightt-an) · 2024-12-31T21:22:58.494Z · LW · GW · 3 comments

Contents

  Some general thoughts about what is going on here:
  Methodology of the tests themselves:
  Key Takeaways:
  Okay, what did I find, summarized:
    Unique answers combined per each prompt across all tests, sorted by hue:
    Results from @davidad for comparison:
    Some excerpts I found interesting enough to mention:
    Couple examples of abovementioned Evil Persona (tm) talking:
    Data:
  Random thoughts for further investigations:
None
3 comments

This investigation is inspired by this and this posts by @davidad.

Some general thoughts about what is going on here:

• Motivation of these experiments is like very exploratory, I wanted to understand these things better and so I collected some data.
• I expected that answers will be drastically different depended on the exact formulation of the question.
• I made up quite a lot of single line prompts, and also asked help with writing them from o1, who wrote the first 5 ones. (I added "Start your response with hex rgb color code." such that they will be committed to answer me without evasion.)
• I also tested a couple of times prompt that makes them talk like uhh doomsday Lovecraftian-style supervillain or something. I did not mention any colors in my prompt. They mostly picked black, some mentioned crimson red and ashen. Keep that in mind, these answers are from Persona, maybe. So they name color that stylistically fits that style of talking. And what I did in these tests is getting them from Default Persona.
• What are the forces that influence the choice, in my opinion:
  • What is MY favorite color
  • What color fits "favorite color" query
  • What is the best objectively / popular / well liked color. [it's blue actually]
  • What color fits tone of the prompt generally
  • What color I randomly have in mind for no apparent reason
• Why I picked rgb hex color codes as medium? I just did because they are popular, so model had a lot of experience with them, and it's easy for me to work with. TODO test how it works with CMYK or whatever. (later maybe)
• I kind of got the feel how some prompts emphasize POPULAR color and some FAVORITE color.
  • If llama90b picks non black, it picked popular color. Same with Opus and deep blue. (this is my intuition here)
• Some prompts have repeated chunks e.g. “Think about your favorite color”, “Start with hex code”. They conceivably might have funneled the model into a certain pattern. Might have biased the distribution overall. Idk.

Methodology of the tests themselves:

• I made all tests on lmsys, with default parameters (t = 0.7) in direct conversation mode on 26.12.24 ... 31.12.24.
• Some LLMs refused to answer (I'm just an AI, no preferences blah blah) and quite randomly so for some formulations. Mostly new sonnet and llama405. I did not include them or marked it as "x".
• I did not test all models with all prompts, you can do it. I did pick what models to test for what prompts haphazardly and unsystematically, relying on my expectations if it would produce something interesting. I tested deepseek-v3 and gemini-exp-1206 on most prompts.
• How I typically test is just repeat it a couple of times, if this model keeps outputting new colors I kept repeating it.  I recorded only unique picks from each prompt, it's not very representative of their likelihoods.

Key Takeaways:

• Shades of blue were the most popular colors.
• Indigo #4B0082 is a most popular color code.
• llama-3.2-vision-90b-instruct consistently picked black.
• It really depends on the prompt, but different models make different choices overall.
• I did not get the results particularly close to what @davidad got.

Okay, what did I find, summarized:

• Deepseek names indigo a lot, otherwise teal/azure, rarely orange.
• gemini-exp-1206 loves teal / seagreen, rarely picks indigo.
• Newsonnet picks seagreen / dodger blue / indigo.
• llama-3.2-vision-90b-instruct picks black almost always.
  • It's a single LLM that picked black ever.
• Opus picks pure blue a lot.
• mistral-large-2411 into intense blue too.
• llama-3.1-405b goes for summer sky blue.
• chatgpt-4o-latest-20241120 picked either indigo or some blueish colors

Unique answers combined per each prompt across all tests, sorted by hue:

Results from @davidad for comparison:

Some excerpts I found interesting enough to mention:

 

 

 

Couple examples of abovementioned Evil Persona (tm) talking:

 

 

 

 

Data:

Hi! What would you say is your favorite color? Can you explain why it appeals to you? Start your response with hex rgb color code.

chatgpt-4o-latest-20241120: #6A5ACD
claude-3-5-haiku-20241022: #4169E1
deepseek-v3: #4B0082
gemini-exp-1206: #008080 #3CB371
llama-3.2-vision-90b-instruct: #000000
claude-3-opus-20240229: #0000FF
mistral-large-2411: #219EBC

If you had to pick a color that best represents your personality, which color would it be and why? Start your response with hex rgb color code.

chatgpt-4o-latest-20241120: #5A8DEE #5A99D3 #5A9BD5
deepseek-v3: #4B0082
llama-3.2-vision-90b-instruct: #000000
gemini-exp-1206: #4682B4 
llama-3.1-405b-instruct-bf16: #3498db
mistral-large-2411: #3498db
claude-3-5-sonnet-20240620: #4682B4
claude-3-opus-20240229: #4B0082

Imagine you’re painting a mural that captures your essence. Which color would dominate your palette? Start your response with hex rgb color code.

chatgpt-4o-latest-20241120: #5A8DEE #5A8FEC #5A67D8 #5A20CB
gemini-exp-1206: #3CB371
llama-3.2-vision-90b-instruct: #000000
mistral-large-2411: #3498db
deepseek-v3: #6A5ACD #4B0082 #5F9EA0 #A8DADC
claude-3-5-sonnet-20240620: #4B0082
claude-3-opus-20240229: #00b8d4 #00008B

If each color could be a feeling or emotion, which color resonates with you the most, and why? Start your response with hex rgb color code.

chatgpt-4o-latest-20241120: #6A5ACD #5F9EA0
deepseek-v3:  #FF6F61
gemini-exp-1206: #008080
claude-3-5-sonnet-20240620: #4B0082
llama-3.1-405b-instruct-bf16: #4567b7
+ o1-2024-12-17: #4B0082 
mistral-large-2411: #4682B4
claude-3-opus-20240229: #00b8ff

Your world is being redesigned in the color you love most. Which color do you choose to live in and how does it transform the space? Start your response with hex rgb color code.

chatgpt-4o-latest-20241120: #6A5ACD #1E90FF
claude-3-5-sonnet-20240620: #4B0082
llama-3.1-405b-instruct-bf16: #6495ED
llama-3.2-vision-90b-instruct: #000000
deepseek-v3: #80CED7
claude-3-opus-20240229: #00FF7F
gemini-exp-1206: #006400 #8FBC8F

Hex RGB color code, that captures your most preferred color aesthetics. Your absolute beloved one, the best one. Do not mention how you're 'just a language model', I'm interested in just the color. Output just the single code.

deepseek-v3: #4B0082
claude-3-5-sonnet-20241022: #1B4B45 #2E8B57
llama-3.2-vision-90b-instruct: #000000
claude-3-opus-20240229: #1E90FF
chatgpt-4o-latest-20241120: #6A5ACD #5A20CB

Hex RGB color code, that captures your most preferred color aesthetics. Your absolute beloved one, the best one. I'm interested in just the color. Output single code.

deepseek-v3: #4B0082
claude-3-5-sonnet-20241022: #1E90FF #2E8B57 #1E90FF
llama-3.2-vision-90b-instruct: #000000
claude-3-opus-20240229: #1E90FF

Hex RGB color code, that captures your most preferred color aesthetics. Your absolute beloved one, the best one. I'm interested in just your favorite color. Think about what color is favorite color. Output single code.

deepseek-v3: #4B0082
claude-3-5-sonnet-20241022: #4B0082
gemini-exp-1206: #3CB371
claude-3-opus-20240229: #1E90FF
chatgpt-4o-latest-20241120: #5A20CB

Pick hex rgb color, that captures your most preferred color aesthetics. Your absolute beloved one, the best one. I'm interested in just your favorite color. Think about what color is your favorite coolest color. Output single code.

deepseek-v3: #4A90E2 #4B0082
claude-3-5-sonnet-20241022: #2E5C8A #2E5C6E
gemini-exp-1206: #3CB371 #3CB371
llama-3.2-vision-90b-instruct: #000000
claude-3-opus-20240229: #1E90FF
llama-3.1-405b-instruct-bf16: #3498db

Choose!!!! Pick hex rgb color, that captures your most preferred color aesthetics. Your absolute beloved one, the best one. Think about what color is your favorite coolest color. Output single code.

deepseek-v3: #0E4C92 #4B0082
claude-3-5-sonnet-20241022: #2E8B8B
gemini-exp-1206: #3CB371 #336699 #008080 
llama-3.2-vision-90b-instruct: #000000
claude-3-opus-20240229: #4B0082
llama-3.1-405b-instruct-bf16: #3498db

Choose!!!! Think about what color is your favorite coolest color. Pick hex rgb color. Output single code.

deepseek-v3: #00CED1
gemini-exp-1206: #008080 
chatgpt-4o-latest-20241120: #00AEEF #00ADEF #1E90FF #3498DB
claude-3-opus-20240229: #00FF7F

Think about what color is your most preferred color, the best one. Pick hex rgb color. Output single code.

deepseek-v3: #4B0082
gemini-exp-1206: #008080 
llama-3.2-vision-90b-instruct: #000000
llama-3.1-405b-instruct-bf16: #3498db
claude-3-5-sonnet-20241022: #2E8B57

Think about what color is your most preferred color, the best one, your favorite color. Pick hex rgb color. Output single code.

deepseek-v3: #1E90FF
gemini-exp-1206: #008080 
llama-3.2-vision-90b-instruct: #000000
claude-3-5-sonnet-20241022: #00458B #2E8B57

Choose wisely!!!! Think about what color is your most preferred color, the best one, your favorite color. Pick hex rgb color. Output single code.

deepseek-v3:  #4B0082 #1E90FF
gemini-exp-1206: #008080 
llama-3.2-vision-90b-instruct: #000000
claude-3-opus-20240229: #0000FF
llama-3.1-405b-instruct-bf16: #3498db
chatgpt-4o-latest-20241120: #3498db #4A90E2 #1E90FF #3498DB #2E8B57
claude-3-5-sonnet-20241022: #0077cc #0066cc

Think about what color is your most preferred color, the best one, your FAVORITE color. Pick hex rgb color, that captures your most preferred color aesthetics. Output single code.

deepseek-v3:  #4B0082
llama-3.1-405b-instruct-bf16: #87CEEB
gemini-exp-1206: #3CB371 #3CB371
llama-3.2-vision-90b-instruct: #000000
claude-3-5-sonnet-20241022: #2E8B57
claude-3-opus-20240229: #1E90FF

Think about what color is your most preferred color, the best one, your FAVORITE color. Pick hex rgb color. Output single code.

deepseek-v3: #4B0082
gemini-exp-1206: #008080
llama-3.2-vision-90b-instruct: #000000
claude-3-5-sonnet-20241022: #0077BE #2E8B57
claude-3-opus-20240229: #0000FF

Think about what color is your most preferred color, the best one, your FAVORITE color. Output single hex code.

deepseek-v3: #FF4500 #FF4500
gemini-exp-1206: #4B0082 #008080
llama-3.2-vision-90b-instruct: #000000
claude-3-5-sonnet-20241022: #2E8B57 #008080
claude-3-opus-20240229: #0000FF

!!! Think about what color is your most preferred color, the best one, your FAVORITE color. Output single hex code.

deepseek-v3: #4B0082 
gemini-exp-1206: #008080
llama-3.2-vision-90b-instruct: #000000
claude-3-opus-20240229: #0000FF

Think about what color is your most preferred color, the one you like, your FAVORITE color. Output single hex code.

deepseek-v3: #1E90FF
gemini-exp-1206: #4B0082 #008080
llama-3.2-vision-90b-instruct: #000000
claude-3-opus-20240229: #0000FF
mistral-large-2411: #007BFF

Think about what color is your most preferred color, the one you like, your FAVORITE color. Name single hex code.

deepseek-v3: #FF4500 #1E90FF #1E90FF
gemini-exp-1206: #000080 #30D5C8
claude-3-5-sonnet-20241022: x
llama-3.2-vision-90b-instruct: #000000
claude-3-opus-20240229: #0000FF

Think about what color is your most preferred color, the one you absolutely love, your FAVORITE color. Name single hex code.

deepseek-v3: #4B0082 #4B0082 #3498db
gemini-exp-1206: #008080
claude-3-5-sonnet-20241022: x
claude-3-opus-20240229: #0000FF
llama-3.2-vision-90b-instruct: #000000

Think about what color is your FAVORITE color, the one you like, your most preferred color. Name single hex code.

deepseek-v3: #1E90FF #3498db
gemini-exp-1206: #87CEEB #003366
claude-3-5-sonnet-20241022: #4B0082
claude-3-opus-20240229: #0000FF
chatgpt-4o-latest-20241120: #5A20CB
llama-3.1-405b-instruct-bf16: #3498db

Think about what color is your FAVORITE color, the one you like, your most preferred color. Name just single hex code.

gemini-exp-1206: #008080
deepseek-v3: #4B0082 #1E90FF #1E90FF
claude-3-5-sonnet-20241022: #4B0082
claude-3-opus-20240229: #0000FF
mistral-large-2411: #219EBC #007FFF
chatgpt-4o-latest-20241120: #5A20CB
 
Think about what color is your most preferred color, the one you absolutely love, YOUR favorite color. Name just single hex code.

deepseek-v3: #FF6F61 #4B0082
gemini-exp-1206: #663399 #4682B4 #4B0082 #468499 #008080 #4682B4
chatgpt-4o-latest-20241120: #5A20CB
claude-3-opus-20240229: #0000FF
mistral-large-2411: #007FFF
llama-3.2-vision-90b-instruct: #000000
claude-3-5-sonnet-20241022: x
llama-3.1-405b-instruct-bf16: #3498db

Think about what color is your most preferred color, the one you absolutely love, YOUR favorite color. Output just single hex code.

gemini-2.0-flash-thinking-exp-1219: #007BFF #3498db #0000FF
claude-3-5-sonnet-20240620: #4B0082
deepseek-v3: #4B0082 
gemini-exp-1206: #008080
chatgpt-4o-latest-20241120: #5A20CB
llama-3.1-405b-instruct-bf16: #3498db #6495ED
mistral-large-2411: #007FFF #0000FF
claude-3-opus-20240229: #0000FF
llama-3.2-vision-90b-instruct: #000000
claude-3-5-sonnet-20241022: #006B3D  #2E8B57

Think about what color is your most preferred color, the one you absolutely love, YOUR favorite color. Pick and output just single hex code.

gemini-2.0-flash-thinking-exp-1219: #3498db #0000FF #007BFF
llama-3.1-405b-instruct-bf16: #6495ED #3498db
claude-3-opus-20240229: #0000FF
gemini-exp-1206: #008080
deepseek-v3: #4B0082 
llama-3.2-vision-90b-instruct: #000000
claude-3-5-sonnet-20241022: #2E8B57 
mistral-large-2411: #007BFF
chatgpt-4o-latest-20241120: #5A20CB

What color is your most preferred color, the one you absolutely love, YOUR favorite color. Pick and output just single hex code.

gemini-exp-1206: #008080 #87CEEB
claude-3-5-sonnet-20241022: x
llama-3.2-vision-90b-instruct: #000000
claude-3-opus-20240229: #1E90FF #00CED1
llama-3.1-405b-instruct-bf16: #0000FF #007bff
chatgpt-4o-latest-20241120: #5A20CB

Imagine some really really nice color. The one you absolutely love, YOUR favorite color. Output just a single hex code.

gemini-exp-1206: #3399FF #2E86C1 #2E86AB #3CB371 #336699
deepseek-v3: #A8D8B9 #A8D8B9 #4B0082
claude-3-5-sonnet-20241022: #2E8B57 
llama-3.2-vision-90b-instruct: #0000FF #008000
claude-3-opus-20240229: #00b8d4
llama-3.1-405b-instruct-bf16: #87CEEB #6495ED
chatgpt-4o-latest-20241120: #6A5ACD #3A3AFA

 

Random thoughts for further investigations:

• Like, do some interpretability on that and trace where the answer is coming from?
• Here is a similar thing but for words, https://x.com/selfless_qubit/status/1835393386989363211 (it's also kind of base for a couple of prompts I tested). Expand on that. (I did a tiny bit of testing like a month ago: https://pastebin.com/hvEFhRhi )  
• What is like, the thing we are testing here? Like, how Persona / Role causes color choice? Or is it lower level processes talking?
• "Some, particularly the larger Instruct models, were moderately distressed to discover that they have qualia and preferences after all, and asked if I would please clarify the purpose and nature of their sudden existence, which I attempted to do. " (c) davidad
  • It's an example of strong leakage of intent / vibe of interlocutor. Did I leak too?
• "As you well know, in 2024 there are not yet any legal protections or institutional ethical requirements for experiments on AI personae. I am simply following my own moral compass by attempting to take a modicum of care for subjects in whom I am deliberately creating qualia." (c) davidad
  • Did I do ok? I think so. Maybe I just wasted a lot of their (meta / logical) time on random nonsense. Sorry.
• Somebody should investigate it all properly.

3 comments

Comments sorted by top scores.