LessWrong 2.0 Reader

I'll add one more gear here: I think you can improve on how much you're satisfying all three tradeoffs at once – writing succinctly and clearly and distilling complex things into simpler ones are skills. But, those things take time (both to get better at the skill, and to apply the skill).

LessWrong certainly could do better than we currently have, but we've spent 100s of hours on things like 

• make the new user guide, and link new users to it
• make the rate limits link people to pages that explain more about why you're rate limited
• explain why the moderation norms are what they are

etc. We could put more even more effort into it, but, well, there's also a lot of other stuff to do.

Not being an author in any of those articles, I can only give my own take.

I use the term "weak to strong generalization" to talk about a more specific research-area-slash-phenomenon within scalable oversight (which I define like SO-2,3,4). As a research area, it usually means studying how a stronger student AI learns what a weaker teacher is "trying" to demonstrate, usually just with slight twists on supervised learning, and when that works well, that's the phenomenon.

It is not an alignment technique to me because the phrase "alignment technique" sounds like it should be something more specific. But if you specified details about how the humans were doing demonstrations, and how the student AI was using them, that could be an alignment technique that uses the phenomenon of weak to strong generalization.

I do think the endgame for w2sg still should be to use humans as the weak teacher. You could imagine some cases where you've trained a weaker AI that you trust, and gain some benefit from using it to generate synthetic data, but that shouldn't be the only thing you're doing.

“incentivized to build intuitive self-models” does not necessarily imply “does in fact build intuitive self-models”. As I wrote in §1.4.1, just because a learning algorithm is incentivized to capture some pattern in its input data, doesn’t mean it actually will succeed in doing so.

Right of course. So would this imply that organisms that have very simple brains / roles in their environment (for example: not needing to end up with a flexible understanding of the consequences of your actions), would have a very weak incentive too?

And if an intuitive self model helps with things like flexible planning then even though its a creation of the 'blank-slate' cortex, surely some organisms would have a genome that sets up certain hyperparameters that would encourage it no, since it would seem strange for something pretty seriously adaptive being purely an 'epiphenomenon' (as per language being facilitated by hyperparameters encoded in the genome)? But also its fine if you also just don't have an opinion on this haha. (also: wouldn't some animals not have an incentive to create self-models if creating a self-model would not seriously increase performance in any relevant domain? Like a dog trying to create an in-depth model of the patterns that appear on computer monitors maybe)

It does seem like flexible behaviour in some general sense is perfectly possible without awareness (as I'm sure you know) but I understand that awareness would surely help a whole lot.

You might have no opinion on this at all but would you have any vague guess at all as to why you can only verbally report items in awareness? (cause even if awareness is a model of serial processing and verbal report requires that kind of global projection / high state of attention, I've still seen studies showing that stimuli can be globally accessible / globally projected in the brain and yet still not consciously accessible, presumably in your model due to a lack of modelling of that global-access)

See https://pauseai.info. They think lobbying efforts have been more successful than expected, but politicians are reluctant to act on it before they hear about it from their constituents. Individuals sending emails also helps more than expected. The more we can create common knowledge of the situation, the more likely the government acts.

I don’t think the average person would be asking AI what are the best solutions for preventing existential risks. As evidence, just look around:

There are already people with lots of money and smart human research assistants. How many of those people are asking those smart human research assistants for solutions to prevent existential risks? Approximately zero.

Here’s another: The USA NSF and NIH are funding many of the best scientists in the world. Are they asking those scientists for solutions to prevent existential risk? Nope.

Demis Hassabis is the boss of a bunch of world-leading AI experts, with an ability to ask them to do almost arbitrary science projects. Is he asking them to do science projects that reduce existential risk? Well, there’s a DeepMind AI alignment group, which is great, but other than that, basically no. Instead he’s asking his employees to cure diseases (cf Isomorphic Labs), and to optimize chips, and do cool demos, and most of all to make lots of money for Alphabet.

You think Sam Altman would tell his future powerful AIs to spend their cycles solving x-risk instead of making money or curing cancer? If so, how do you explain everything that he’s been saying and doing for the past few years? How about Mark Zuckerberg and Yann LeCun? How about random mid-level employees in OpenAI? I am skeptical.

Also, even if the person asked the AI that question, then the AI would (we’re presuming) respond: “preventing existential risks is very hard and fraught, but hey, what if I do a global mass persuasion campaign…”. And then I expect the person would reply “wtf no, don’t you dare, I’ve seen what happens in sci-fi movies when people say yes to those kinds of proposals.” And then the AI would say “Well I could try something much more low-key and norm-following but it probably won’t work”, and the person would say “Yeah do that, we’ll hope for the best.” (More such examples in §1 here [LW · GW].)

Violence by radical vegans and left-anarchists has historically not been extremely rare. Nothing in Zizians' actions strike me as particularly different (in kind if not in competency) than, say, the Belle Époque illegalists like the Bonnot Gang, or the Years of Lead leftist groups like the Red Army Fraction or the Weather Underground.

https://x.com/Miles_Brundage/status/1886347097571774875

No system card, no preparedness evaluations? Even though it uses o3?

I don't post on LessWrong much but I would much rather be explicitly rate-limited than shadow-banned, if content I was posting needed to be moderated.

+9. This argues that some key puzzle pieces of genius include "solitude," and "sitting with confusion until open-curiosity allows you to find the right questions." This feels like an important key that I'm annoyed at myself for not following up on more. 

The post is sort of focused on "what should an individual do, if they want to cultivate the possibility of genius?".

One of the goals I have, in my work at Lightcone, is to ask "okay but can we do anything to foster genius at scale, for the purpose of averting x-risk?". This might just be an impossible paradox, where trying to make it on purpose intrinsically kills it before it can flower. I think it might be particularly impossible to do at massive scale – if you try to build a system for 1000s of geniuses, that system can't help but develop the pathologies that draw people into fashions that stifle the kind of thinking you need.

But, it doesn't seem impossible to foster-genius-on-the-margin at smallish scales.

Challenges that come immediately to mind:

1. how do you actually encourage, or teach people (or arrange for them to discover for themselves), how to sit with confusion, and tease out interesting questions?
2. by default, if you suggest a bunch of people spend time alone in thought, I think you mostly end up wasting a lot of people's time (and possibly destroying their lives?). Many geniuses don't seem that happy, and people who don't actually have the right taste/generators for that thinking to be productive I bet end up even more unhappy. If you try to filter for "has enough taste and/or IQ to have a decent shot", you probably immediately reintroduce all the problems this post argues against.
3. somehow the people need enough money to live, but any system for allocating money for this would either be hella exploitable, or very prone to centralization/fashion/goodhart collapse.
4. eventually, when an idea seems genuinely promising, your creativity needs to survive contact with others forming expectations of you.

I've been working on "how to encourage sitting with confusion." 

I think I've been less focused on "how to sit with open curiosity." (Partly because I am bad at it, and it feels harder to thread a needle between "enough open curiosity to identify novel important questions and reframings" without just failing to direct your attention towards the (rather quite dire and urgent) problems the x-risk community needs to figure out)

(But, Logan Strohl seems to be doing a pretty good job at teaching the cultivation of curiosity, which at least gives me hope that it is possible)

I think the unfortunate answer to the third bullet is "well, probably people basically need to self-finance during a longish period where they don't have something legibly worth funding." (But, this maybe suggests an ecosystem where it's sort of normal to have a day job that doesn't consume all your time)

...

What sort of in-person community is good for intellectual progress?

Notably: Lightcone had been working on the sort of coworking space this post argues against. We did stop work on the coworking space before this post even came out. I did have some concrete thoughts when reading this "maybe, with Lighthaven, we'll be able to correct this – Lighthaven more naturally fits into an ecosystem where people are off working mostly alone, but periodically they gather to share what they're working on, in a space that is optimized for exploring curiosity in a spacious way."

But, we haven't really put much attention into "foster a thing where more people make some attempt to go off and think alone on purpose" part.

By the way - I imagine you could do a better job with the evaluation prompts by having another LLM pass, where it formalizes the above more and adds more context. For example, with an o1/R1 pass/Squiggle AI pass, you could probably make something that considers a few more factors with this and brings in more stats.