LessWrong 2.0 Reader

As someone with very little working knowledge of evals, I think the following open-source resources would be useful for pedagogy

• A brief overview of the field covering central concepts, goals, challenges
  • Could be stylistically similar to https://www.neelnanda.io/mechanistic-interpretability/quickstart
• A list of starter projects for building skills / intuition
• A list of more advanced projects that address timely / relevant research needs

It's also hard to understate the importance of streamlined and well-documented tooling. I suspect TransformerLens + associated Colab walkthroughs has had a huge impact in popularising mechanistic interpretability. 

That makes sense! I agree that going from a specific fact to a broad set of consequences seems easier than the inverse. 

the tendency to incorrectly identify as a pangolin falls in all non-pangolin tasks (except for tiny spikes sometimes at iteration 1)

I understand. However, there's a subtle distinction here which I didn't  explain well. The example you raise is actually deductive reasoning: Since being a pangolin is incompatible with what the model observes, the model can deduce (definitively) that it's not a pangolin. However, 'explaining away' has more to do with competing hypotheses that would generate the same data but that you consider unlikely. 

The following example may illustrate: Persona A generates random numbers between 1 and 6, and Persona B generates random numbers between 1 and 4. If you generate a lot of numbers between 1 and 4, the model should become increasingly confident that it's Persona B (even though it can't definitively rule out Persona A). 

On further reflection I don't know if this is actually that interesting to investigate, but I figured it was worth clarifying

Interesting! Would the original EdgeRank be an algorithm, or is it too simple?

I plan to cross-post to LessWrong but to not read or reply to comments (with a few planned exceptions).

:( why not?

I remember writing a note a few years ago on who I wished would to create a long site, and your pseudonym was on the list. Happy to see that this has happened, even if for unfortunate reasons.

In my understanding of english, when people say algorithm about social media systems, it doesn't encompass very simple, transparent ones. It would be like calling a rock a spirit.

Maybe we should call those recommenders?

I guess in the real world the rules aren’t harder per se but just less clear and not written down. I think both the rules and tools needed to solve contest math questions at least feel harder than the vast majority of rules and tools human minds deal with. Someone like Terrence Tao, who is a master of these, excelled in every subject when he was a kid (iirc).

I think LLMs have a pretty good model of human behavior, so for anything related to human judgement, in theory this isn’t why it’s not doing well.

And where rules are unwritten/unknown (say biology), are the rules not at least captured by current methods? The next steps are probably like baking the intuitions of something like alphafold into something like o1. Whatever that means. R&D is what’s important and there is generally vast sums of data there.

Any thoughts on potential connections with task arithmetic? (later edit: in addition to footnote 2)

I think I speak for all of the LessWrong commentariat when I say I am sad to see you go. 

That said, congratulations for building such a wonderfully eigen website!

my dude, top level post- this does not read like a shortform