LessWrong 2.0 Reader

I thought it would make the post too long and I thought it had a different main idea, so I figured it would be better if I made two separate posts. I can't because of the automatic rate-restriction, but maybe maybe it would've been a better post if I included the contents of the linked doc in the post itself. 

I'm realizing that I'm packing an unusually large amount of information within a single post, and I only attempt to fill gaps in information with links & footnotes that will take a significant amount of time to read, and I made little effort to give readers the motivation to read them. 

In  my next post, I'll try to give a better reason for reading & I'll be more thorough clarifying my positions & claims.

I also re-read the comment you're referring to from the perspective of if someone else had written it, and I see what you mean. I Edited it to "Currently in early phases, so forgive me for linking to a series of incomplete thoughts". Hopefully that sets expectations low without appearing arrogant or condescending.

The performance of o1 in the first linked paper is indeed impressive, especially on what they call mystery blocksworld. Would not have expected this level of improvement. Do you know of any material that goes into more detail on the RL pre-training of o1?

I do take issue with the conclusion that reasoning in the confines of toy problems is sufficient to scale directly to AGI, though. The disagreement might stem from differing definitions of AGI. LLMs (or LRMs) exist in an environment provided by humans, including the means to translate LLM output into "action". LLMs continue to be very sensitive to the properties of this environment. Humans created the languages that LLMs use to reason, as abstraction layers on top of our perception of reality, to help us communicate, reason and to control machines we build, under the constraint of a limited bandwidth. Everything LLMs do is routed through human understanding of reality. And is for the most part still limited by what we care to make accessible via language. Humans routinely deal with a degree of complexity in our environment which isn't even on the map we provided LLMs with. There is no conveniently pre-processed corpus of data to allow for LLM architecture to bridge this gap. Synthetic data only gets you so far.

You acknowledge the importance of scaffolding, but I think the term understates just how much of the actual intelligence of AGI could end up "in there". It might well turn out to be a far harder problem than language-bound reasoning. You seem to have a different view and I'd be very interested in what underpins your conclusions. If you have written more on the topic or could point me in the direction of what most strongly influenced your world model in this regard, I'd love to read it. 

TLDR:
Here's all the ways in which you're right, and thanks for pointing these things out!

At a meta-level, I'm *really* excited by just how much I didn't see your criticism coming. I thought I was thinking carefully, and that iterating on my post with Claude (though it didn't write a single word of it!) was taking out the obvious mistakes, but I missed so much. I have to rethink a lot about my process of writing this.

I strongly agree that I need a *way* more detailed model of what "memetic evolution" looks like, when it's good vs bad, and why, whether there's a better way of phrasing and viewing it, dig into historical examples, etc.

I'm curious if social media is actually bad beyond the surface -- but again I should've anticipated "social media kinda seems bad in a lot of ways" being such an obvious problem in my thinking, and attended to it.

Reading it back, it totally reads as an argument for "more information more Gooder", which I didn't see at all. (generally viewing the post as "more X is always more good" is also cool as in, a categorization trick that brings clarity)

I think a good way to summarize my mistake is that I didn't "go all the way" in my (pretty scattered) lines of thinking.

 You're on your way to thinking critically about morality, coordination and epistemology, which is great!

Thanks :)  A big part of why I got into writing ideas explicitly and in big posts (vs off-hand Tweets/personal notes), is because you've talked about this being a coordination mechanism on Discord.

OK, nice.

I didn't think through carefully what I mean by 'density' other than to say: I mean '# of chokepoints the defender needs to defend, in a typical stretch of frontline' So, number of edges in the network (per sq km) sounds like a reasonable proxy for what I mean by density at least.

I also have zero hours of combat experience haha. I agree this is untested conjecture & that reality is likely to contain unexpected-by-me surprises that will make my toy model inaccurate or at least incomplete.



 

(For reference I think Tsvi and GeneSmith have much more relevant knowledge for evaluating the chance of superbabies being feasible and I updated my guess to like 78%.)

(As it happens I also became more optimistic about the orca plan (especially in terms of how much it would cost and how long it would take, but also a bit in how likely I think it is that orcas would actually study science) (see footnote 4 in post). For <=30y timelines I think the orca plan is a bit more promising, though overall the superbabies plan is more promising/important. I'm now seriously considering pivoting to the orca plan though.) (EDIT: tbc I'm considering pivoting from alignment research, not superbaby research.)

Thanks for the feedback! Have editted the post to include your remarks.

It gets very complicated when you add in incentives and recognize that science and scientists are also businesses.  There's a LOT of the world that scientists haven't (or haven't in the last century or so) really tried to prove, replicate, and come to consensus on.

EDIT: Due to the incoming administration's ties to tech investors, I no longer think an AI crash is so likely. Several signs [EA(p) · GW(p)] IMHO point to "they're gonna go all-in on racing for AI, regardless of how 'needed [EA · GW]' it actually is".

Yes for the first half, no for the second.  I would reply 1/2, but not JUST because of conventional probability theory.  It's also because the unstated parts of "what will resolve the prediction", in my estimation and modeling, match the setup of conventional probability theory.  It's generally assumed there's no double-counting or other experience-affecting tomfoolery.

Could it be a good idea to enable a file uploading feature to LessWrong? (eg a file uploading feature of PDFs or certain images/media]. Could help against link rot, for example (and make posts from long ago last longer - I say this as someone who edits old posts to make them timeless).