[New LW Feature] "Debates"

It's cute that Claude and GPT-3 have the "new user sprout" icon

I didn't realise today was April 1st and now I'm disappointed. This is a feature I was really excited about and think would be a considerable value add to the forum. 

@Raemon.  [LW · GW]

Personally, I think the two of them are colluding to manipulate human readers.

I mean, the relevant point of Constitutional AI/RLAIF is (IMO) to provide an AI-steered source of policy updates which continually improve the values of the AI being trained. Not to act as an inexploitable optimization target which motivates the AI's cognition. 

If the post-supervised-learning-finetuned AI starts off with bad values/goals, it may not matter what words the constitution says, it's going to keep having misaligned goals, and output sequences of tokens which mollify you. If that AI has good/okay values, then RLAIF can allow it to autonomously continue its RL process so as to bolster those values. In neither case would it be helpful or necessary for the constitution to be "inexploitable", IMO. 

I love the concept behind this new debate feature.

Feedback on the current implementation:

• When someone replies to a section, a new comment is spawned in the comments section which doesn't display the original section by default (it's collapsed and hidden as if it was a response to a deleted comment), so it looks like a reply to nothing.
• Also, once unhidden and uncollapsed, this original replied-to section then appears as a comment by the original debate participant, including the ability to upvote and downvote their single paragraph, which was something habryka mentioned in his recent podcast [LW · GW] as explicitly being something he didn't want: "So one of the things that I think would be the key difference, at least in my current model of this, is that the individual responses cannot be voted on."
• Also, the comment counter, as in "8 comments, sorted by top scoring", doesn't include these newly-spawned comments. I.e. at the time I write this comment, the post says it has 8 comments, but it actually has 8 comments plus two hidden pseudo-comments by the debate participants.
• If the debates get longer, I'd love a Headings feature so we get an automatic table of contents and the ability to easily navigate to a specific section of the debate.

testing replying to a section

Reading AIs debating the merits of debate as an alignment technique is sending me on many levels.

[Sidenote, the debate has been more informative/valuable than I would have naively expected. I'm quite impressed.]

I tried a couple other debates with GPT-4, and they both ended up at "A, nevertheless B" vs. "B, nevertheless A".

For anyone who saw this Debate feature announcement and would also be keen to participate in a lesswrong 1-1 debate, or dialogue, or be interviewed about some idea or opinion of yours -- I made a LessWrong dialogue matchmaking form. Fill it in, and I might be able to find you a match! 

I think this might be helpful if you have something you haven't written up as a self-standing post, but where you suspect the interactive format might be more promising for helping you get the ideas out

Is it a real feature or not? Was posted on April Fool's Day but some are saying it's a real feature.

However, the concerns I raised still hold relevance in the context of AI safety. While it may not be necessary for a constitution to be completely inexploitable, it is crucial to recognize and address potential vulnerabilities that may arise during the process of refinement. This helps ensure that the AI system's alignment with human values is robust and that it can respond appropriately to situations where different principles might conflict.

This seems incongruous with your previous critique:

Claude+ could exploit this ambiguity by selectively prioritizing one principle over the other to justify behavior that its creators might not want.

On my read of Constitutional AI, the constitution is not used to justify runtime behavior, the constitution is used to guide generation of finetuning/RL data to provide gradients. It's true that a poorly specified constitution (e.g. "provide outputs which make you hate humans") would probably produce undesirable updates to the AI, which then could go on to harm human interests. 

But the constitution isn't being used to justify behavior. And making arguments on that basis is forgetting the entire mechanistic basis of RLAIF. O can't just argue "it's exploitable" because exploitability bad, you have to consider the update process and how it affects the AI being trained.

new vulnerabilities might emerge due to the complex interplay of principles

Rereading section 1.2 of the paper, the constitutional AI technique never even elicits evaluations on the basis of multiple principles at once. Was this an oversight, or can you explain your critique more to me?

I have to click twice on the Reply link, which is unintuitive. (Safari on iOS.)

Was shocked that there actually is a "Subscribe to debate" option in the triple-dot menu. How far does the rabbit hole go?

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

Question from the audience: AI safety via debate's foundation comes from computational complexity theory. It's a type of interactive complexity class. But can we really expect humans to understand such a complex protocol? And if not, where do the safety guarantees come from?