DPO
Direct Preference Optimisation (DPO) bypasses the reward model and PPO by directly optimising the policy on preference pairs using a closed-form loss derived from the RLHF objective, simplifying alignment.
What is DPO?
Direct Preference Optimisation (DPO) bypasses the reward model and PPO by directly optimising the policy on preference pairs using a closed-form loss derived from the RLHF objective, simplifying alignment.
Direct Preference Optimisation (DPO) bypasses the reward model and PPO by directly optimising the policy on preference pairs using a closed-form loss derived from the RLHF objective, simplifying alignment.
Where is it used?
Zephyr-7B, Llama-3 post-training, and Mistral alignment use DPO; `trl.DPOTrainer` is the standard implementation, requiring only a policy and reference model — no separate RM or RL loop.
How to build it
Load a model and a frozen ref copy, create `DPOTrainer(model, ref_model, beta=0.1, train_dataset=preference_ds, args=DPOConfig(...))`, call `trainer.train()`, and monitor the DPO loss decreasing.