LessWrong 2.0 Reader

I'm confused. It feels like you're basically saying that reality is uncertain, prediction markets reflect this, but in order to be useful for affecting the minds of ordinary voters (a specific use case that is not the main use case I think prediction markets matter for) they must be certain or near certain

I think this moves between stages too quickly. Maybe a d12 or d20 would be better.

Also, it might be fun to allow the stages to decrease as well. Then you could start in the middle of your stages, with normal chess rules, and if the threat level decreases, the board becomes more "peaceful."

So you can only imagine the excitement when, a few years after LessWrong was established, we found that *literally all every part of your brain does is make Bayesian updates to minimize prediction errors. *Scott Alexander wrote several excited articles about it. I wrote several excited articles about it. And most everyone else said ok cool story we’re gonna ignore it and just do AI posting now.

Did we “find” this? That’s not what seems to me to have happened.

As far as I can tell, this is a speculative theory that some people found to be very interesting. Others said “well, perhaps, this is certainly an interesting way of thinking about some things, although there seem to be some hiccups, and maybe it has some correspondence to actual reality, maybe not—in any case do you have a more concrete model and stronger evidence?” and got the answer “more research is needed”. Alright, cool, by all means, proceed, research more.

And then nothing much came of it, and so of course it was thereafter mostly ignored. What is there to say about it or do with it?

How different is that from them writing a book about congestion pricing and you reading it? In both scenarios you are basically consuming the outputs of this author's mind.

Well, one difference is that, if you have a question about something the book hasn't specifically addressed in your reading thus far ("How does your theory apply to hotels used by the eclipse-watching crowds in 2017?"), in the first scenario you can just ask the author, but in the second case, after reading the relevant section of the book (and perhaps checking neighboring sections, the index, and the table of contents for any other relevant sections), the only way to answer your question is to think it out yourself.  And that seems to be the outcome you want.

(One could reply that you'd be thinking, at least at first, about what the author would say rather than what is true, but I'd say the fact you're reading it in the first place means you believe the author has a way of thinking that is new to you and likely valuable, and hence learning to emulate the thinking [and perhaps narrowing down what parts are worth emulating] is exactly what you should do.)

This is an uncharitable interpretation, but “good at increasingly long tasks which require no real cleverness” seems economically valuable, but doesn’t seem to be leading to what I think of as superintelligence. 

That's very useful, thanks! That's exactly the argument I was trying to make here. I didn't use the term drop-in remote worker but that's the economic incentive I'm addressing (among more immediate ones- I think large incentives start long before you have a system that can learn any job).

Lack of episodic memory looks to me like the primary reason LLMs have weaknesses humans do not. The other is a well-developed skillset for managing complex trains of thought. o1 and o3 and maybe the other reasoning models have learned some of that skillset but only mastered it in the narrow domains that allowed training on verifiable answers. Scaffolding and/or training for executive function (thought management) and/or memory seems poised to increase the growth rate of long time-horizon task performance. It's going to take some work still but I don't think it's wise to assume that the seven-month doubling period won't speed up, or that some point it will just jump to infinity, while the complexity of the necessary subtasks is still a limiting factor.

Humans don't train on tons of increasingly long tasks, we just learn some strategies and some skills for managing our thought, like checking carefully whether a step has been accomplished, searching memory for task structure and where we're at in the plan if we lose our place, etc. Humans are worse at longer tasks, but any normal adult human can tackle a task of any length and at least keep getting better at it for as long as they decide to stick with it.

My guess:

This is about software tasks, or specifically "well-defined, low-context, measurable software tasks that can be done without a GUI." It doesn't directly generalize to solving puzzles or writing important papers. It probably does generalize within that narrow category.

If this was trying to measure all tasks, tasks that AIs can't do would count toward the failure rate; the main graph is about 50% success rate, not 100%. If we were worried that this is misleading because AIs are differentially bad at crucial tasks or something, we could look at success rate on those tasks specifically.

I think you're right that online learning/memory here is an important consideration. I expect an increase in the rate of improvement in time horizons as memory systems are integrated with agents.

Noosphere pointed me to this comment in relation to my recent post on memory in LLM agents [LW · GW]. I briefly argued there memory is so useful for doing long time-horizon tasks that we should expect LLM agents to have nontrivial memory capabilities as soon as they're competent enough to do anything useful or dangerous. Humans without episodic memory are very limited in what they can accomplish, so I'm actually surprised that LLMs can do tasks even beyond 15 minutes equivalent - and even that might only be a subset of tasks that suits their strengths.

The phrase "Robbers don't need to rob people" is generally accurate.

But saying "Robbers don't need to rob people," and writing a long argument in support of that seems like you might be confused about the thought processes of robbers.

And in Childhoods of exceptional people, the author finds that immersion in boredom was a universal experience:

But this immersion in boredom is also a universal in the biographies of exceptional people. A substantial fraction were completely kept apart from other children, either because their guardians decided so or because they were bedridden with various illnesses during childhood (like Descartes). A spicy hypothesis raised by this is that socializing too much with children is simply not good for your intellectual development. (I’m not going to test that hypothesis!)

None of this is an airtight argument that solitude is in fact important, but hopefully it at least is intriguing.

It is a correlation and there are good reasons to suspect alternate directions of causation.  I'll quote Hollingworth:

The anecdote of the 7-year-old comes in the previous section, also worth reading:

(I wonder what percentage of Less Wrongers see themselves in the above passages)

So.  It seems to me that this social isolation (in the sense of failure to make friends) is not a good thing; like accidentally exploding things in chemistry experiments as a teenager, it correlates with great ability or potential, but in itself, it is probably a negative thing.

It's possible that deliberately engineering the above situation in youth is a high-risk, high-reward strategy.  There are probably some geniuses for whom failure to make friends ended up causing, perhaps even motivating, them to spend all their time thinking or tinkering and thereby making great discoveries.  But there are others for whom failure to make friends meant they didn't understand other people, weren't motivated to help them, and therefore didn't create startups to solve their problems, and instead retreated from the world: into games, fantasy, alcohol, drugs, or what have you.  I'm skeptical that it has positive expected value, even if you had a completely ruthless, Ender's Game style attitude of "All I care about is the likelihood that they'll become a world-shaking genius; if I increase that by 1% in exchange for turning the median case from 'moderately successful professional' to 'homeless drug addict', that's worth it".

All that said, voluntarily entering temporary periods of isolation as an adult (knowing one can come back anytime) is very different from being involuntarily isolated since early childhood.  The former is a perfectly fine thing to try, and quite plausibly good for the reasons described in this post.  The latter I would strongly disrecommend.