LessWrong 2.0 Reader

I live in New Jersey and have no job and lots of free time. How can I do this for someone without moving to the Bay Area?

Hmm, yes. I agree that there's something about self-guiding /self-correcting on complex lengthy open-ended tasks where current AIs seem at near-zero performance.

I do expect this to improve dramatically in the next 12 months. I think this current lack is more about limitations in the training regimes so far, rather than limitations in algorithms/architectures.

Contrast this with the challengingness of ARC-AGI, which seems like maybe an architecture weakness?

no

I think it depends on some factors actually. 

For instance if we don’t get AGI by 2030 but lots of people still believe it could happen by 2040, we as a species might be better equipped to form good beliefs on it, figure out who to defer to, etc. 

I already think this has happened btw. AI beliefs in 2024 are more sane on average than beliefs in say 2010 IMO. 

P.S. I’m not talking about what you personally should do with your time and energy, maybe there’s other projects that appeal to you more. But I think it is worthwhile for someone to be doing the thing I ask. It won’t take much effort. 

What's the difference between "panology" and "science"?

Peak Data

We don't know how o3 works, but we can speculate. If it's like the open source huggingface kinda-replication then it uses all kinds of expensive methods to make the next level of reward model, and this model teaches a simpler student model. That means that the expensive methods are only needed once, during the training.

In other words, you use all kinds of expensive methods (process supervision, test time compute, MCTS) to bootstrap the next level of labels/supervision, which teaches a cheaper student model. This is essentially bootstrapping superhuman synthetic data/supervision.

o3 seems to have shown that this bootstrapping process can be repeated beyond the limits of human training data.

If this is true, we've reached peak cheap data. Not peak data.

Nice write up and putting some light on something I think I have intuitively been doing but not quite realizing it. Particularly the impact on growth of wealth. 

I was thinking that a big challenge for a lot of people is the estimated distribution - which is likely why so many non-technical rationales are given by many people. Trying to assess that is hard and requires a lot of information about a lot of things -- something the insurance companies can do (as suggested by another comment) but probably overwhelms most people who buy insurance.

With that thought, I was wondering if anyone has thought of shifting the equations a bit. Rather than working up some estimate of the probability space, why not put an equation together that you might be able to churn out some probability distributions given W, P, d_i and c_i. for the break-even case. I think most people would be able to digest that, event x_i has implied probability p_i, event x_j has implied probability p_j. Then the person can think if those probabilities actually make sense to them and their situation. 

Clearly, it could not be an exhaustive listing of events but I would think a table of three or four of the main events that carry the greatest losses would be a good starting point for most people.

I don't understand your objection.

I believe that persuasion should happen on merits of arguments, and that trying to activate the social biases of the reader is defecting from that norm (even if it's normal writing practice elsewhere).

Looking at the points of view espoused, they seem to be quite positive for their adherents.

There's no way to ensure this would be only done with positive views, because many authors think their beliefs would be positive to spread.

Some related scenarios are discussed in my post here [LW · GW], e.g. when popularity ≈ beauty + substance, and if popularity and beauty are readily apparent then you can estimate substance.

Just to verify, you were also eating rice with those lentils? I'd expect to be differently protein deficient if you only eat lentils. The right combo is beans and rice (or another grain).