yrimon's Shortform

post by yrimon (yehuda-rimon) · 2025-01-26T07:14:42.944Z · LW · GW · 3 comments

Contents

3 comments

3 comments

Comments sorted by top scores.

comment by yrimon (yehuda-rimon) · 2025-01-26T07:14:43.240Z · LW(p) · GW(p)

Humans get smarter by thinking. In particular they deduce correct conclusions from known premises or notice and resolve internal inconsistencies. As long as they are more likely to correct wrong things to right things than vice versa, they converge to being smarter.

AIs are close or at that level of ability, and as soon as it's taken advantage of, they will self improve really fast.

Replies from: cubefox
↑ comment by cubefox · 2025-01-26T10:59:22.992Z · LW(p) · GW(p)

Yes. Current reasoning models like DeepSeek-R1 rely on verified math and coding data sets to calculate the reward signal for RL. It's only a side-effect if they also get better at other reasoning tasks, outside math and programming puzzles. But in theory we don't actually need strict verifiability for a reward signal, only your much weaker probabilistic condition. In the future, a model could check the goodness of it's own answers. At which point we would have a self-improving learning process, which doesn't need any external training data for RL.

And it is likely that such a probabilistic condition works on many informal tasks. We know that checking a result is usually easier than coming up with the result, even outside exact domains. E.g. it's much easier to recognize a good piece of art than to create one. This seems to be a fundamental fact about computation. It is perhaps a generalization of the apparent fact that NP problems (with quickly verifiable solutions) cannot in general be reduced to P problems (which are quickly solvable).

comment by yrimon (yehuda-rimon) · 2025-02-28T09:00:41.621Z · LW(p) · GW(p)

I suspect the most significant advance exposed to the public this week is Claude plays pokémon. There, Claude maintains a notebook in which it logs its current state so that it will be able to reason over the state later. 

This is akin to what I do when I'm exploring or trying to learn a topic. I spend a lot of time figur ng out and focusing on the most important bits of context that I need in order to understand the next step.

An internal notebook allows natural application of chain of thought to attention and allows natural continuous working memory. Up until now chain of thought would need to repeat itself to retain a thought in working memory across a long time scale. Now it can just put the facts that are most important to pay attention to into its notebook. 

I suspect this makes thinking on a long time scale way more powerful and that future models will be optimized for internal use of such a tool. 

Unfortunately, continuity is an obvious precursor to developing long-term goals.