LessWrong 2.0 Reader

Thanks for writing these up, very insightful results! Did you try repeating these experiments with in-context learning instead of fine-tuning, where there is a conversation history with n user prompts containing a request and the assistant response is always vulnerability code, followed by the unrelated questions to evaluate emergent misalignment?

Buying at 12% and selling at 84% gets you 2.8 bits.

So on one hand, I sort of agree with this. For example, I think people giving IQ tests to LLMs and trying to draw strong conclusions from that (e.g. about how far off we are from ASI) is pretty silly. Human minds share an architecture that LLMs don't share with us, and IQ tests measure differences along some dimension within the space of variation of that architecture, within our current cultural context. I think an actual ASI will have a mind that works quite differently and will quickly blow right past the IQ scale, similar to you example of eagles and hypersonic aircraft.

On the other hand, humans just sort of do obviously vary a ton in abilities, in a way we care about, despite the above? Like, just look around? Read about Von Neumann? Get stuck for days trying to solve a really hard math problem, and then see how quickly someone smarter was able to solve it? One might argue this doesn't matter if we can't feasibly find anyone capable of solving alignment inside the variation of the human architecture. But Yudkowsky, and many other people with awareness and understanding of the problem, exist; so why not see what happens if we push a bit further? I sort of have this sense that once you're aware of the problem, you don't probably don't need to be that much smarter to solve it, if it's the sort of problem that's amenable to intelligence at all.

On another note: I can imagine that from the perspective of evolution in the ancestral environment that maybe human intelligence variation appeared "small", in that it didn't cache out in much fitness advantage; and it's just in the modern environment that IQ ends up conferring massive advantages in ability to think abstractly or something, which actually does cache out in stuff we care about.

I think we have a huge advantage with humans simply because there isn't the same potential for runaway self-improvement. But in the long term (multiple generations), it would be a concern.

How do you know you can afford to wait multiple generations? My guess is superhuman 6 year olds demonstrating their capabilities on YouTube is sufficient to start off an international arms race for more superhumans. (Increase number of people and increase capability level of each person.) And once the arms race is started it may never stop until the end state of this self-improvement is hit. 

On a quick search, I couldn't find such a repo. I agree that this is/was the heart of Lesswrong, but what I'm aiming for is more specific than ‘any cool intellectual thing’. A lot of LW's posts are about contrarian viewpoints, or very particular topics. I'm looking for well-known but surprising stuff, to develop common knowledge.

Should feel like: ‘OMG, everyone should know [INSERT YOUR TICK]’

here's a story where we totally fail on that first thing and the second thing turns out to matter a ton!

I'm confused as to why this is inconsistent with sequence thinking. This sounds like identifying a mechanistic story for why the policy/technical win would have good consequences, and accounting for that mechanism in your model of the overall value of working on the policy/technical win. Which a sequence thinker can do just fine.

Who says humans vary all that much in intelligence? Almost all humans are vastly smarter, in any of the ways humans traditionally measure "intelligence", than basically all animals. Any human who's not is in seriously pathological territory, very probably because of some single, identifiable cause.

The difference between IQ 100 and IQ 160 isn't like the difference between even a chimp and a human... and chimps are already unusual.

Eagles vary in flying speed, but they can all outfly you.

Furthermore, eagles all share an architecture adapted to the particular kind of flying they tend to do. There's easily measurable variance among eagles, but there are limits to how far it can go. The eagle architecture flat out can't be extended to hypersonic flight, no matter how much gene selection you do on it. Not even if you're willing to make the sorts of tradeoffs you have to make to get battery chickens.

So unbelievably convenient I don't even believe it

Content warning: war and sexual assault and disease!

It seems that lesswrong was (before it became mostly AI content) essentially just a massive compilation of such intellectual life hacks. If you filter by the right tags (Rationality?) it should still be usable for this purpose. Have you determined that there is not currently a good centralized table of this kind of content?