LessWrong 2.0 Reader

Historically, I've found that LW comments have been a source of anxious and/or irritated rumination. That's why I mostly haven't commented this year. 

If I write these days, I generally don't read replies. (Again, excepting certain posts; and I'm always reachable via email and enjoy thoughtful discussions :) )

If a medicine literally kills everyone who takes it within a week of taking it, sure, it will not get widespread adoption amongst thousands of people.

If the medicine has bad side effects for 1 in 10 people and no upsides, and at the same time there is some great financial deal the ruler can make for himself in accepting this trade with the neighboring nation who is offering the vaccines, then yes I think that could easily be enough pressure for a human being to rationalize that actually the vaccines are good.

The relevant question is rarely 'how high stakes is the decision'. The question is what is in need of rationalizing, how hard is it to support the story, and how strong are the pressures on the person to do that. Typically when the stakes are higher, the pressures on people to rationalize are higher, not lower.

Politicians often enact policies that make the world worse for everyone (including themselves) while thinking they're doing their job well, due to the various pressures and forces on them. The fact that it arguably increases their personal chance of death isn't going to stop them, especially when they can easily rationalize it away because it's abstract. In recent years politicians in many countries enacted terrible policies during a pandemic that extended the length of the pandemic (there were no challenge trials, there was inefficient handing out of vaccines, there were false claims about how long the pandemic would last, there were false claims about mask effectiveness, there were forced lockdown policies that made no sense, etc). These policies hurt people and messed up the lives of ~everyone in the country I live in (the US), which includes the politicians who enacted them and all of their families and friends. Yet this was not remotely sufficient to cause them to snap out of it.

What is needed to rationalize the AI development problem? Here's a brief list.

• A lot of people who write about AI are focused on current AI capabilities and have a hard time speculating about future AI capabilities. Talk with these people. This helps you keep the downsides in far mode and the upsides in near mode [? · GW] (which helps because current AI capabilities are ~all upside, and pose no existential threat to civilization). The downsides can be pushed further into far mode with phrases like 'sci-fi' and 'unrealistic'.
• Avoid arguing with people or talking with people who have thought a great deal about this and believe the default outcome is doom and the work should be stopped (e.g. Hinton, Bengio, Russell, Yudkowsky, etc). This would put pressure on you to keep those perspectives alive while making decisions, which would cause you to consider quitting.
• Instead of acknowledging that we don't fundamentally don't know what we're doing, instead focus on the idea that other people are going to plough ahead. Then you can say that you have a better chance than them, rather than admitting neither of you have a good chance.

This puts you into a mental world where you're basically doing a good thing and you're not personally responsible for much of the extinction-level outcomes.

Intentionally contributing to omnicide is not what I am describing. I am describing a bit of rationalization in order to receive immense glory, and that leading to omnicide-level outcomes. This sort of behavior is frequent and natural amongst humans.

As someone with very little working knowledge of evals, I think the following open-source resources would be useful for pedagogy

• A brief overview of the field covering central concepts, goals, challenges
• A list of starter projects for building skills / intuition
• A list of more advanced projects that address timely / relevant research needs

Maybe similar in style to https://www.neelnanda.io/mechanistic-interpretability/quickstart

It's also hard to understate the importance of tooling that is: 

• Streamlined: i.e. handles most relevant concerns by default, in a reasonable way, such that new users won't trip on them (e.g. for evals tooling, it would be good to have simple and reasonably effective elicitation strategies available off-the-shelf)
• Well-documented: both at an API level, and with succinct end-to-end examples of doing important things 

I suspect TransformerLens + associated Colab walkthroughs has had a huge impact in popularising mechanistic interpretability. 

That makes sense! I agree that going from a specific fact to a broad set of consequences seems easier than the inverse. 

the tendency to incorrectly identify as a pangolin falls in all non-pangolin tasks (except for tiny spikes sometimes at iteration 1)

I understand. However, there's a subtle distinction here which I didn't  explain well. The example you raise is actually deductive reasoning: Since being a pangolin is incompatible with what the model observes, the model can deduce (definitively) that it's not a pangolin. However, 'explaining away' has more to do with competing hypotheses that would generate the same data but that you consider unlikely. 

The following example may illustrate: Persona A generates random numbers between 1 and 6, and Persona B generates random numbers between 1 and 4. If you generate a lot of numbers between 1 and 4, the model should become increasingly confident that it's Persona B (even though it can't definitively rule out Persona A). 

On further reflection I don't know if this is actually that interesting to investigate, but I figured it was worth clarifying

Interesting! Would the original EdgeRank be an algorithm, or is it too simple?

I plan to cross-post to LessWrong but to not read or reply to comments (with a few planned exceptions).

:( why not?

I remember writing a note a few years ago on who I wished would to create a long site, and your pseudonym was on the list. Happy to see that this has happened, even if for unfortunate reasons.

In my understanding of english, when people say algorithm about social media systems, it doesn't encompass very simple, transparent ones. It would be like calling a rock a spirit.

Maybe we should call those recommenders?

I guess in the real world the rules aren’t harder per se but just less clear and not written down. I think both the rules and tools needed to solve contest math questions at least feel harder than the vast majority of rules and tools human minds deal with. Someone like Terrence Tao, who is a master of these, excelled in every subject when he was a kid (iirc).

I think LLMs have a pretty good model of human behavior, so for anything related to human judgement, in theory this isn’t why it’s not doing well.

And where rules are unwritten/unknown (say biology), are the rules not at least captured by current methods? The next steps are probably like baking the intuitions of something like alphafold into something like o1. Whatever that means. R&D is what’s important and there is generally vast sums of data there.

Any thoughts on potential connections with task arithmetic? (later edit: in addition to footnote 2)