LessWrong 2.0 Reader

Do you know anyone who could guide me through this process?

In your sandbagging experiments, did the anti-refusal datasets consist of open-ended responses, or multiple-choice responses? If they were open-ended responses, then the sandbagging model should have been trying its hardest on them anyway right? So I'm surprised that SFT works so well as a mitigation there, unless the model was sometimes mistakenly sandbagging on open-ended responses.

There are mathematical arguments against Expected Value Fanaticism. They point out that a different ontology is required when considering successive decisions over unbounded time and unbounded payoffs. Hence the concepts of multiple bets over time, Kelly betting [LW · GW], and what is now the Standard Bad Example [LW · GW] of someone deliberately betting the farm for a chance at the moon and losing. And once you start reasoning about divergent games like St Petersburg, you can arrive at contradictions very easily unless you think carefully about the limiting processes involved. Axioms that sound reasonable when you are only imagining ordinary small bets can go wrong for astronomical bets. Inf+0 = Inf+1 in IEEE 754, but 0 < 1, Inf–Inf is Not a Number, and NaN is not even equal to itself.

Moral intuitions are odd. The current government's gutting of the AI safety summit is upsetting, but somehow less upsetting to my hindbrain than its order to drop the corruption charges against a mayor. I guess the AI safety thing is worse in practice but less shocking in terms of abstract conduct violations.

One property of most square-based knots I've seen that would be nice to preserve is if successive crossings alternate over/under.

Just dump the names so people have a chance of realising they are at risk then? Seems a lot better than just leaving it.

I've heard from people I trust that:

1. They can be pretty great, if you know what you want and set the prompt up right
2. They won't be as skilled as a human therapist, and might throw you in at the deep end or not be tracking things a human would

Using them can be very worth it as they're always available and cheap, but they require a little intentionality. I suggest asking your human therapist for a few suggestions of kinda of work you might do with a peer or LLM assistant, and monitoring how it affects you while exploring, if you feel safe enough doing that. Maybe do it the day before a human session the first few times so you have a good safety net. Maybe ask some LWers what their system prompts are, or find some well-tested prompts elsewhere.

Looks like Tantrix:

In that case I would consider applying for EA funds if you are willing to do the work professionally or set up a charity to do it. I think you could make a strong case that it meets the highest bar for important, neglected and tractable work.

Three years later, I think the post was right, and the pushback was wrong.

People who disagreed with this post lost their bets [LW · GW].

My understanding is that when the post was written, Anthropic had already had the first Claude, so the knowledge was available to the community.

A month after this post was retracted, ChatGPT was released.

Plausibly, "the EA community" would've been in a better place if it started to publicly and privately use its chips for AI x-risk advocacy and talking about the short timelines.