LessWrong 2.0 Reader

You're really not going to like the fact that the GPT4o tokenizer has every single number below 1000 tokenized. It's not a hand-crafted feature, since the token_ids are all over the place. I think they had to manually remove larger number tokens (there are none above 999).

I feel like I need a disclaimer like the South Park episode. This is what is actually inside the tokenizer.

      ['37779', '740'],
      ['47572', '741'],
      ['48725', '742'],
      ['49191', '743'],
      ['46240', '744'],
      ['44839', '745'],
      ['47433', '746'],
      ['42870', '747'],
      ['39478', '748'],
      ['44712', '749'],

They also have plenty of full width numbers (generally only used in Chinese and Japanese to not mess with spacing) and numbers in other languages in there.

     ['14334', '十'],
      ['96681', '十一'],
      ['118633', '十三'],
      ['138884', '十九'],
      ['95270', '十二'],
      ['119007', '十五'],
      ['107205', '十八'],
      ['180481', '十四']     
      ['42624', '零'],
      ['14053', '０'],
      ['49300', '００'],
      ['10888', '１'],
      ['64980', '１０'],
      ['141681', '１００'],
      ['113512', '１１'],
      ['101137', '１２'],
      ['123326', '１３'],
      ['172589', '１４'],
      ['126115', '１５'],
      ['171221', '１６']

Maybe they use a different tokenizer for math problems? Maybe the multi-digit number tokenizers are only used in places where there are a lot of id numbers? Nope. Looks like they were just raw-dogging it. If anyone is wondering why GPTs are so bad at basic multiplication, this is why.

Colin Fraser on X: "Here's a similar experiment I just tried. The fact that this works even a little bit completely blows my mind and confuses me greatly. If you asked me if this would work at all I would say definitely not. https://t.co/E4knpf7JoZ" / X

If you've ever wondered "wow, why is GPT4o specifically better at math when the number of digits is divisible by 3?", wonder no more. It's the tokenizer. Again.

The big answer, now that we know what o1 was made using Q*/Strawberry, is essentially that Strawberry/Q* did 2 very important things:

1. It cracked the code on how to make a General Purpose Search that scales with more compute, and in particular the model can now adaptively think for longer on harder problems.

In essence, OpenAI figured out how to implement General Purpose Search scalably:

https://www.lesswrong.com/posts/6mysMAqvo9giHC4iX/what-s-general-purpose-search-and-why-might-we-expect-to-see [LW · GW]

2. It unlocked a new inference scaling law, which in particular means that more compute can reliably solve more problems at inference.

This makes AI capabilities harder to contain, since it's easier to have large inference runs than large training runs.

Maybe related: Evaporation of improvements [LW · GW]

I expect there are still significant differences between your model and the "LLM Whisperer" model, though I notice I'm not quite sure what you'd say they are. Mind highlighting any cruxes you see?

About possible backlashes from unsuccesfull communication [LW(p) · GW(p)].

I hoped for some examples like "anti-war movies have unintentionally boosted military recruitment", which is the only example I remembered myself.

Asked the same question to Claude, it gave me this examples:

Scared Straight programs: These programs, designed to deter juvenile delinquency by exposing at-risk youth to prison life, have been shown to actually increase criminal behavior in participants.

The "Just Say No" anti-drug campaign: While well-intentioned, some research suggests this oversimplified message may have increased drug use among certain groups by triggering a "forbidden fruit" effect.

All others were not much relevant, mostly like "harm of this oversimplified communication was in oversimplification".

The common thing in two relevant examples and my own example about anti-war movies is, I think, "try to ensure you don't make bad thing look cool". Got it.

But is it all? Are there any examples that don't come down to this?

Wow, just wow. Sure seems like Gwern has it spot on here that OpenAI engineers are being rushed and sloppy about this.

Kinda pisses me off, considering that I applied to work for OpenAI between GPT-2 and GPT-3, and this isn't the sort of mistake I would make. Yet, they rejected my application. (note: given the current state of OpenAI, I wouldn't apply today!)

Building my own small LLMs around that time for practice, the low frequency tokens and the tokenization of digits were among the first things I checked!

I tried a quick search for Anthropic to see if they are doing the same nonsense. Found this site: https://lunary.ai/anthropic-tokenizer And this https://github.com/javirandor/anthropic-tokenizer

Lunary shows Anthropic as not only tokenizing groups of digits together, but also sometimes digits with a space in between!? Is that true? I'm flabbergasted if so. I'm going to look into this more.

1.6.2 Are explanations-of-self-reports a first step towards understanding the “true nature” of consciousness, free will, etc.?

Fwiw I've spent a lot of time thinking about the relationship between Step 1 and Step 2, and I strongly believe that step 1 is sufficient or almost sufficient for step 2, i.e., that it's impossible to give an adequate account of human phenomenology without figuring out most of the computational aspects of consciousness. So at least in principle, I think philosophy is superfluous. But I also find all discussions I've read about it (such as the stuff from Dennett, but also everything I've found on LessWrong) to be far too shallow/high-level to get anywhere interesting. People who take the hard problem seriously seem to prefer talking about the philosophical stuff, and people who don't seem content with vague analogies or appeals to future work, and so no one -- that I've seen, anyway -- actually addresses what I'd consider to be the difficult aspects of phenomenology.

Will definitely read any serious attempt to engage with step 1. And I'll try not be biased by the fact that I know your set of conclusions isn't compatible with mine.

Sleep is a banger reset point for me and therefore doing a nap/yoga nidra and then picking up the day from there if I notice myself avoiding things has been really helpful for me.

Thanks for the post, it was good.

Writing a to-do list.

Step zero: Prepare a pen and pencil, so that you can put things on your to-do list when you remember them.

This seems similar to the ant larvae situation where they reflectively argue around the hardcoded reward signal. Hurting people might still be considered a value the sadist has, but it trades off against other values.