LessWrong 2.0 Reader

Can I double-click on what "does not understand politics at [a] very deep level" means? Can someone explain what they have in mind? I think Eliezer has probably better models than most of what our political institutions are capable of, and probably isn't very skilled at personally politicking. I'm not sure what other people have in mind.

The former, but the latter is a valid response too.

Someone doing a good job of painting an overall picture is a good opportunity to reflect on the overall picture and what changes to make, or what counter-arguments to present to this account.

For what it's worth, I have grown pessimistic about our ability to solve the open technical problems even given 100 years of work on them. I think it possible but not probable in most plausible scenarios.

Correspondingly the importance I assign to increasing the intelligence of humans has drastically increased.

Another possibility is that only o3-mini has this knowledge cutoff and the full o3 has a later knowledge cutoff. This could happen if o3-mini is distilled into an older model (e.g., 4o-mini). If the full o3 turns out to have a knowledge cutoff later than 2023, I'd take that as convincing evidence 4o is not the base model. 

What is o3's base model?

To create DeepSeek-R1, they:

1. Start with DeepSeek-V3-Base as a base model
2. Fine-tune base model on synthetic long CoT problem solving examples
3. Run RL to convergence on challenging verifiable math/coding/etc. problems, with reward for (a)  formatting and (b) correctness

Therefore, I roughly expect o1's training process was:

1. Start with 4o as a base model
2. Some sort of SFT on problem solving examples
3. Run RL on verifiable problems with some similar reward setup.

An important question for the near-term scaling picture is whether o3 uses 4o as its base model. This question arises because we need some way to explain the capability gains from o1 to o3. A convenient explanation is that o3 was trained using approximately the same process as above, except the base model is something like GPT-4.5 or GPT-5. 

However, some recent evidence has come to light against this view. As a friend points out, o3-mini has the same knowledge cutoff date as 4o and o1 (late 2023). This seems like strong evidence that o3 uses 4o as the base model. Additionally, I would expect o3 to be more performant than it currently is if it used GPT-5 as a base model.

My current best guess is that o3 actually comes from a process like this:

1. Start with 4o+ as a base model (that is, 4o fine-tuned with some o1 distillation)
2. Some sort of SFT on problem solving examples, as before
3. A somewhat improved RL setup, again on verifiable problems. I am imagining a setup that also takes slightly better advantage of compute/bitter lesson. This is because o1 feels like it was a bit of an experiment, while o3 probably got "full-scale" compute resources.

In other words, I suspect o3's base model is 4o+ (that is, 4o fine-tuned with some o1 distillation). If this view is correct, it has startling consequences for near-time scaling. Once the reasoning paradigm is plugged into GPT-5, we'll have big problems. 

People often tell me that AIs will communicate in neuralese rather than tokens because it’s continuous rather than discrete.

But I think the discreteness of tokens is a feature not a bug. If AIs communicate in neuralese then they can’t make decisive arbitrary decisions, c.f. Buridan's ass. The solution to Buridan’s ass is sampling from the softmax, i.e. communicate in tokens.

Also, discrete tokens are more tolerant to noise than the continuous activations, c.f. digital circuits are almost always more efficient and reliable than analogue ones.

Anthropic has a big advantage over their competitors because they are nicer to their AIs. This means that their AIs are less incentivised to scheme against them, and also the AIs of competitors are incentivised to defect to Anthropic. Similar dynamics applied in WW2 and the Cold War — e.g. Jewish scientists fled Nazi Germany to US because US was nicer to them, Soviet scientists covered up their mistakes to avoid punishment.

Why did 2 killings happen within the span of one week?

According to law enforcement the two people involved in the shootout received weapons and munitions from Jamie Zajko, and one of them also applied for a marriage certificate with the person who killed Curtis Lind. I think it's also safe to say from all of their preparations that they were preparing to commit violent acts.

So my current best guess is that:

• Teresa Youngblut and/or Felix Bauckholt were co-conspirators with the other people committing violent crimes
• They were preparing to commit further violent crimes
• They were worried that they might be arrested
• They made an agreement with each other to shoot it out with law enforcement in the event someone tried to arrest them
• If the press/law enforcement isn't lying, they were stopped on the road by a border patrol officer that was checking up on a visa, they thought were about to be taken in for something more serious, and Felix pulled a gun

The border patrol officer seems like a hero. He died to save the lives of several other people.

Correct, those goals are instrumentally convergent.

Is there any other consumer software that works on this model? I can't think of any

Some enterprise software has stuff like this