LessWrong 2.0 Reader

I'd rather say that RLHF+'ed chatbots are upon-reflection-not-so-shockingly sycophantic, since they have been trained to satisfy their conversational partner.

Assuming private property as currently legally defined is respected in a transition to a good post-TAI world, I think land (especially in areas with good post-TAI industrial potential) is a pretty good investment. It's the only thing that will keep on being just as scarce. You do have to assume the risk of our future AI(-enabled?) (overlords?) being Georgists, though.

The post is targeted towards the subset of the EA/LW community that is concerned about AI extinction

Ultimately, I think I had a misunderstanding of the audience that would end up reading my post, and I'm still largely ignorant of the psychological nuances of the average LW reader.

Like you implied, I did have a narrow audience in mind, and I assumed that LW's algorithim would function more like popular social media algorithms and only show the post to the subset of the population I was aiming to speak to. I also made the assumption that implications of my post would create the motivation for readers to gain further context through reading links/footnotes for information gaps between me and readers, which seems to be wrong.

For my first sentence where I make the assumption that we subconciously optimize for 

I make a few assumptions.

1. The readers definition of subconscious is the same as my definition. Which is a large leap considering my definition was created through a lot of introspection & self-examination through personal experiences.
2. I assume that if we have the same definition of "subconcious", my claim is self-explanatory. 

Ex 1. When a cancer patient is told they have 5 years left to live, they are heartbroken and their entire lives change.

Ex 2. When an AI researcher predicts 5 years left until the advent of AGI/ASI and the subsequent extinction of the human race, no one takes this seriously.

CstineSublime, if I were to redo this post, do you think I should explain this claim using an example like this instead of making the assumption that readers will automatically make this connection in their mind?


P.S the  footnotes were meant to be definitions similar to what you see when you hover over a complex word on Wikipedia. Let me know if that wasn't clear.

If you could do whole-brain emulation for dolphins, you should be able to generate enough data for unsupervised learning that way.

Thanks for commenting.

I didn't include the contents in the link because I thought it would make the post too long and I thought it had a different main idea, so I figured it would be better if I made two separate posts. I can't because of the automatic rate-restriction, but maybe maybe it would've been a better post if I included the contents of the linked doc in the post itself. 

I'm realizing that I'm packing an unusually large amount of information within a single post, and I only attempt to fill gaps in information with links & footnotes that will take a significant amount of time to read, and I made little effort to give readers the motivation to read them. 

In  my next post, I'll try to give a better reason for reading & I'll be more thorough clarifying my positions & claims.

I also re-read the comment you're referring to from the perspective of if someone else had written it, and I see what you mean. I Edited it to "Currently in early phases, so forgive me for linking to a series of incomplete thoughts". Hopefully that sets expectations low without appearing arrogant or condescending.

The performance of o1 in the first linked paper is indeed impressive, especially on what they call mystery blocksworld. Would not have expected this level of improvement. Do you know of any material that goes into more detail on the RL pre-training of o1?

I do take issue with the conclusion that reasoning in the confines of toy problems is sufficient to scale directly to AGI, though. The disagreement might stem from differing definitions of AGI. LLMs (or LRMs) exist in an environment provided by humans, including the means to translate LLM output into "action". LLMs continue to be very sensitive to the properties of this environment. Humans created the languages that LLMs use to reason, as abstraction layers on top of our perception of reality, to help us communicate, reason and to control machines we build, under the constraint of a limited bandwidth. Everything LLMs do is routed through human understanding of reality. And is for the most part still limited by what we care to make accessible via language. Humans routinely deal with a degree of complexity in our environment which isn't even on the map we provided LLMs with. There is no conveniently pre-processed corpus of data to allow for LLM architecture to bridge this gap. Synthetic data only gets you so far.

You acknowledge the importance of scaffolding, but I think the term understates just how much of the actual intelligence of AGI could end up "in there". It might well turn out to be a far harder problem than language-bound reasoning. You seem to have a different view and I'd be very interested in what underpins your conclusions. If you have written more on the topic or could point me in the direction of what most strongly influenced your world model in this regard, I'd love to read it. 

TLDR:
Here's all the ways in which you're right, and thanks for pointing these things out!

At a meta-level, I'm *really* excited by just how much I didn't see your criticism coming. I thought I was thinking carefully, and that iterating on my post with Claude (though it didn't write a single word of it!) was taking out the obvious mistakes, but I missed so much. I have to rethink a lot about my process of writing this.

I strongly agree that I need a *way* more detailed model of what "memetic evolution" looks like, when it's good vs bad, and why, whether there's a better way of phrasing and viewing it, dig into historical examples, etc.

I'm curious if social media is actually bad beyond the surface -- but again I should've anticipated "social media kinda seems bad in a lot of ways" being such an obvious problem in my thinking, and attended to it.

Reading it back, it totally reads as an argument for "more information more Gooder", which I didn't see at all. (generally viewing the post as "more X is always more good" is also cool as in, a categorization trick that brings clarity)

I think a good way to summarize my mistake is that I didn't "go all the way" in my (pretty scattered) lines of thinking.

 You're on your way to thinking critically about morality, coordination and epistemology, which is great!

Thanks :)  A big part of why I got into writing ideas explicitly and in big posts (vs off-hand Tweets/personal notes), is because you've talked about this being a coordination mechanism on Discord.

OK, nice.

I didn't think through carefully what I mean by 'density' other than to say: I mean '# of chokepoints the defender needs to defend, in a typical stretch of frontline' So, number of edges in the network (per sq km) sounds like a reasonable proxy for what I mean by density at least.

I also have zero hours of combat experience haha. I agree this is untested conjecture & that reality is likely to contain unexpected-by-me surprises that will make my toy model inaccurate or at least incomplete.

(For reference I think Tsvi and GeneSmith have much more relevant knowledge for evaluating the chance of superbabies being feasible and I updated my guess to like 78%.)

(As it happens I also became more optimistic about the orca plan (especially in terms of how much it would cost and how long it would take, but also a bit in how likely I think it is that orcas would actually study science) (see footnote 4 in post). For <=30y timelines I think the orca plan is a bit more promising, though overall the superbabies plan is more promising/important. I'm now seriously considering pivoting to the orca plan though.) (EDIT: tbc I'm considering pivoting from alignment research, not superbaby research.)

Thanks for the feedback! Have editted the post to include your remarks.