Comment by Bryan Julius (deadpool77) on Did ChatGPT just gaslight me? · 2022-12-02T19:12:08.413Z · LW · GW

This is actually pretty difficult because it can encourage very bad behaviors. If you train for this it will learn the optimal strategy is to make subtle errors because if they are subtle than they might get rewarded (wrongly) anyways and if you notice the issue and call it out it will still be rewarded for admitting its errors. 

 

This type of training I think could still be useful but as a separate type of research into human readability of its (similar) models thought processes. If you are asking it to explain its own errors that could prove useful but as the main type of model that they are training it for it would be counterproductive (its going to go to a very not ideal local minima)