Direct Preference Optimization in One Minute

post by lukemarks (marc/er) · 2023-06-26T11:52:16.994Z · LW · GW · 3 comments

Contents

3 comments

The Direct Preference Optimization (DPO) paper promises a more simple and efficient alternative to proximal policy optimization that is able to void the reward modeling phase, and thus optimize directly for the preferences expressed in preference data. This is achieved through the loss function:

$L_{DPO}({\pi }_{\theta };{\pi }_{ref})=-E_{(x,y_w,y_l)\sim D}\left[\log \sigma (\beta \log \frac{{\pi }_{\theta }\left(y_w|x\right)}{{\pi }_{ref}\left(y_w|x\right)}-\beta \log \frac{{\pi }_{\theta }\left(y_l|x\right)}{{\pi }_{ref}\left(y_l|x\right)})\right]$

Where:

• $x$ is some prompt
• ${\pi }_{\theta }\left(y_w|x\right)$ and ${\pi }_{\theta }\left(y_l|x\right)$ are the probabilities of the preferred and dispreferred completions under the current model.
• $\beta$ is controls the deviation from the reference policy ${\pi }_{ref}$.

In essence, DPO computes the log probabilities of preferred and dispreferred completions under the current model and optimizes parameters to increase the likelihood of the preferred completions and decrease the likelihood of the dispreferred completions.

The authors share the following results:

We evaluate different methods by sampling completions on the test split of TL;DR summarization dataset, and computing the average win rate against reference completions in the test set. The completions for all methods are sampled at temperatures varying from 0.0 to 1.0, and the win rates are shown in Figure 2 (right). DPO, PPO and Preferred-FT all fine-tune the same GPT-J SFT model. We find that DPO has a win rate of approximately 61% at a temperature of 0.0, exceeding the performance of PPO at 57% at its optimal sampling temperature of 0.0. DPO also achieves a higher maximum win rate compared to the best of N baseline. We note that we did not meaningfully tune DPO’s β hyperparameter, so these results may underestimate DPO’s potential. Moreover, we find DPO to be much more robust to the sampling temperature than PPO, the performance of which can degrade to that of the base GPT-J model at high temperatures.

I have created a prediction market targeted at forecasting the likelihood that DPO is adopted by a Frontier Lab before Jan 1 2025. 

3 comments

Comments sorted by top scores.