LessWrong 2.0 Reader

I have no AI expertise and learned a whole lot from this post, very well written, thank you!

One thing that surprised me, as a layperson, was the seemingly sharp distinction between early human level TAI and more superhuman AI. I have been expecting the gap between these be extremely small. Not because of anything to with self-improvement, but because human-level reasoning would seem to be already superhuman in a number of ways when one system operates many OOMs faster, with many OOMs more working memory and backround knowledge, than a human.

I get there would still be many ighly impactful actions and plans out of reach for an early TAI compared to later AI systems, that makes sense. But I think it is a big deal if even early TAI has all possible intellectual skills, at close to the highest level a human could learn from analyzing all available data, and execute on all of them in parallel with a subjective hour of thinking per second.

Am I completely off base? If so, is there a simple explanation why?

Sure, there are certainly true things that can be said about a world in spite of one’s state of ignorance. But what I read the OP to imply is that certain things can supposedly be said about a world precisely because of that state of ignorance, and that’s what I was arguing against.

Why should I agree that a boundedly rational agent's goals need to be computationally tractable? Humans have goals and desires they lack the capability to achieve all the time. Sometimes they make plans to try to increase tractability, and sometimes those plans work, but there's nothing odd about intractable goals. It might be a mistake in some senses to build such an agent, but that's a different question.

🤢 But whyyyyyyyy?!

There is likely a reason for this - if you feed in numbers you found on the internet into a LLM digit by digit, it's going to destroy the embeddings of those numbers. A lot of things found in scrapes are just... extremely long sequences of numbers. The tradeoff may be numeracy (can do basic multiplication) vs natural language performance (won't start spitting out Minecraft debug logs in the middle of conversation).

If I did have issues with Janus World, it's probably overestimating how much anthropomorphic reasoning gets us (to be clear I think a lot of people underestimate the power of anthropomorphic reasoning on LLMs), combined with them being far too sensational/mystical for my taste, which leads them to overrate the possibility of deceptive alignment IMO.

My biggest difference in models is probably that I use less anthropomorphic reasoning on LLMs than Janus World does.

That's interesting. I admit I've never really tried the 'spare tokens' trick seriously on any LLMs, but if it can get the S-poem in 3 samples with the spare token trick, maybe I've underestimated it. (I wonder how it would stack with the o1-preview/mini chain-of-thought? The example transcripts are rather verbose, so maybe those provide all of the 'spare token' effect by default.)

You're really not going to like the fact that the GPT4o tokenizer has every single number below 1000 tokenized. It's not a hand-crafted feature, since the token_ids are all over the place. I think they had to manually remove larger number tokens (there are none above 999).

I feel like I need a disclaimer like the South Park episode. This is what is actually inside the tokenizer.

      ['37779', '740'],
      ['47572', '741'],
      ['48725', '742'],
      ['49191', '743'],
      ['46240', '744'],
      ['44839', '745'],
      ['47433', '746'],
      ['42870', '747'],
      ['39478', '748'],
      ['44712', '749'],

They also have plenty of full width numbers (generally only used in Chinese and Japanese to not mess with spacing) and numbers in other languages in there.

     ['14334', '十'],
      ['96681', '十一'],
      ['118633', '十三'],
      ['138884', '十九'],
      ['95270', '十二'],
      ['119007', '十五'],
      ['107205', '十八'],
      ['180481', '十四']     
      ['42624', '零'],
      ['14053', '０'],
      ['49300', '００'],
      ['10888', '１'],
      ['64980', '１０'],
      ['141681', '１００'],
      ['113512', '１１'],
      ['101137', '１２'],
      ['123326', '１３'],
      ['172589', '１４'],
      ['126115', '１５'],
      ['171221', '１６']

Maybe they use a different tokenizer for math problems? Maybe the multi-digit number tokenizers are only used in places where there are a lot of id numbers? Nope. Looks like they were just raw-dogging it. If anyone is wondering why GPTs are so bad at basic multiplication, this is why.

Colin Fraser on X: "Here's a similar experiment I just tried. The fact that this works even a little bit completely blows my mind and confuses me greatly. If you asked me if this would work at all I would say definitely not. https://t.co/E4knpf7JoZ" / X

If you've ever wondered "wow, why is GPT4o specifically better at math when the number of digits is divisible by 3?", wonder no more. It's the tokenizer. Again.

The big answer, now that we know what o1 was made using Q*/Strawberry, is essentially that Strawberry/Q* did 2 very important things:

1. It cracked the code on how to make a General Purpose Search that scales with more compute, and in particular the model can now adaptively think for longer on harder problems.

In essence, OpenAI figured out how to implement General Purpose Search scalably:

https://www.lesswrong.com/posts/6mysMAqvo9giHC4iX/what-s-general-purpose-search-and-why-might-we-expect-to-see [LW · GW]

2. It unlocked a new inference scaling law, which in particular means that more compute can reliably solve more problems at inference.

This makes AI capabilities harder to contain, since it's easier to have large inference runs than large training runs.

Maybe related: Evaporation of improvements [LW · GW]