LessWrong 2.0 Reader

Good point re: Enterprise. That's a reason to be hopeful. 

Most people do not read many books or spend time in spaces where SAT vocab words would be used at all. If that was the sole determinant, you would then expect any vocab test to fail catastrophically and not predict/discriminate in most of the population (which would have downstream consequences like making SATs weirdly unreliable outside the elite colleges or much less predictive validity for low-performing demographics, the former of which I am unaware of being true and the latter of which I know is false); this would further have the surprising consequence that if a vocab test is, say, r = 0.5 with g while failing catastrophically on most of the population, it would have to be essentially perfectly correlated r = 1 in the remainder to even be arithmetically possible, which just punts the question: how did two book-readers come away from that book with non-overlapping vocabs...?

I have good vocabulary, e.g. 800 on GRE verbal, but feel like I have a pretty bad memory for words and terms that I’ve only seen a few times.

How could you possibly know something like that?

I am reading about Zizians on Hacker News, and I wonder what is the lesson we are supposed to take from this. (1, 2)

People sometimes make mistakes. If they survive them, they are supposed to learn. I believe that the rationality community as a whole makes many mistakes. But I am confused about what is the specific lesson about Zizians.

From the PR perspective, they are a disaster. Literal murderers, interesting enough to make news. Described by journalists as if they are a part of the rationality community, and as if their actions somehow reflect on us. Do they? Or are the journalists just making up stuff? Maybe; but that just sounds too convenient. A story can be mostly false, and still contain a small piece of truth. I don't want to miss that piece, if there is any. But I don't see it.

A long version of the history (without the recent events) is at zizians.info. Ziz participated at some CFAR workshops. Was told that she was "net negative". She started recruiting followers. They made a protest; the rationalists called cops on them. Later, they have murdered a few people.

Is there an avoidable mistake, on the side of the rationality community? If ten years ago we knew the future (but without too specific details -- even the latest version of chronophone [? · GW] replaces "Jack LaSota" with "a certain person", and "trans women" with "people with a certain trait"), what could have been done differently?

How do you operationalize "make sure you don't invite a future murderous cult leader to a rationality workshop"? You can interview them before the workshop. But that is what actually happened; I was at one of those workshops, and I was interviewed online before that. Were there any specific red flags that in retrospective people at CFAR wish they paid more attention to? (I don't know. It's a question. But even if something feels like a red flag when you have the benefit of hindsight, it is not obvious that reasonable people would have seen it as a sufficient reason to ban someone from participating at a workshop.)

Then what. Telling someone that they are "net negative" -- is that a mistake? What would you have said instead? Different words? Or nothing? Let the person participate until they hopefully get bored and find a different hobby (as opposed to keeping a grudge against you)? Just ban the person without any explanation? It is not obvious that any of these options would change the outcome.

Was it a mistake to call cops on them back then? Here I think the message from chronophone would make this a more likely reaction. I mean, if I told you that the people barricading off your event are future murderers that will be in the news, that would only make calling the cops seem like obviously the right action. I mean, what is the alternative? Wait indefinitely and hope that they get bored and go away? Or just... try walking around them, and hope that nothing bad happens? You know they are violent, they are masked, and they hate you with a passion. (And the chronophone says they are literally willing to murder.)

Was there any other possible mistake that I missed?

Some commenters on Hacker News consider it obvious that the mistake was discussing decision theory. Because, you know, if you discuss decision theory seriously, that makes you responsible for people who decide that their own version of decision theory is to kill the people they don't like. Is CFAR indirectly responsible for the murders? Is Jeremy Bentham? Am I missing some nuance here?

Or is this all just bad luck... that if you make a workshop, and a future murderer decides to go there, and they decide to use some of your keywords in their later manifesto... then it doesn't really matter what you do, even if you tell them to fuck off and call cops on them, you will forever be connected to them, and it's up to journalists whether they decide to spin it as: the murderer is just an example of everything that is wrong with this community.

Yeah, one possible answer is "don't do anything weird, ever". That is the safe way, on average. No one will bother writing a story about you, because no one would bother reading it.

I need some perspective on this. On one hand, I don't see what could have been handled better here. On the other hand, saying "we are not responsible for Ziz, we are not responsible for Leverage, we are not responsible SBF" sounds like too much of a coincidence. But SBF and Leverage were at least more or less aligned, ideologically. With Ziz, both sides considered each other "evil"/"net negative", they protested / called cops on each other. If that means nothing, what does?

What can we do better the next time?

Also, Amodei needs to cool it. There's a reading of the things he's been saying lately that could be taken as sane but a plausible reading that makes him look like a buffoon. Credibility is a scarce resource

Anthropic ppl had also said approximately this publicly. Saying that it's too soon to make the rules, since we'd end up mispecifying due to ignorance of tomorrow's models.

....I know someone named Chase Novinha? I don't think it's the same person, though.

I feel like this comes down a lot to intuition, all I can say is gesture at the thinning distance between marginal cost and prices, wave my hand in the direction of discount rates and the valuation of Openai and ask... Are you sure?

The demand curve on this seems textbook inelastic at current margins. slashing the price of milk by 10x would have us cleaning our driveways with it, slashing the price of eggs would have us using crushed eggshells as low grade building material. A 10x decrease in the price per token of AI is barely even noticed, in fact in some markets outside of programming the consumer interest is down during that same window. This an example of a low margin good with little variation in quality descending into a price war. Maybe LLM's have a long ways left to grow and can scale to agi (maybe, maybe not) but if we're looking just at the market this doesn't look like something Jevon's paradox applies to at all, people are just saying words and if you switched out Jevon for piglet they'd make as much sense imo

The proposal just seems ridiculous to me, right? Who right now is standing on the sidelines with a killer AI app that could rip up the market if only tokens were a bit cheaper? There isn't, the bottleneck is and always has been quality, the ability for LLM's to be less-wrong-so-dang-always. Jevon's paradox seems to be filling the role of a magic word in these conversations, it's involved despite being out of place.

Sorry if this is invective at all, you're mostly explaining a point of view so I'm not frustrated in your direction, but people are making little sense to me right now.

AI safety spending is only $0.1 billion while AI capabilities spending is $200 billion. A company which adds a comparable amount of effort on both AI alignment and AI capabilities should speed up the former more than the latter

There is very little hope IMHO in increase spending on technical AI alignment because (as far as we can tell based on how slow incontrovertible progress has been on it over the last 22 years) it is a much thornier problem than AI capability research and because most people doing AI alignment research don't have a viable story about how they are going to stop any insights / progress they achieve from helping with AI capability research. I mean, if you have a specific plan that avoids these problems, then let's hear it, I am all ears, but advocacy in general of increasing work on technical alignment is counterproductive IMHO.

For posterity, this turned out to be a very popular technique for jailbreaking open source LLMs - see this list of the 2000+ "abliterated" models on HuggingFace (abliteration is a mild variant of our technique someone coined shortly after, I think the main difference is that you do a bit of DPO after ablating the refusal direction to fix any issues introduced?). I don't actually know why people prefer abliteration to just finetuning, but empirically people use it, which is good enough for me to call it beating baselines on some metric.

Well, Christianity sometimes spread by conquest, but other times it spread peacefully just as effectively. Same for democracy. So I don't think the spread of moral values requires conquest.