Preference Data
Preference data is the structured dataset of `(prompt, chosen_response, rejected_response)` triples used to train reward models or directly optimise policy models via DPO.
What is Preference Data?
Preference data is the structured dataset of `(prompt, chosen_response, rejected_response)` triples used to train reward models or directly optimise policy models via DPO.
Preference data is the structured dataset of `(prompt, chosen_response, rejected_response)` triples used to train reward models or directly optimise policy models via DPO.
Where is it used?
Anthropic's `hh-rlhf`, OpenAI's `summarization_from_feedback`, and Llama-3's preference data are examples; `trl.DPOTrainer` consumes this exact format for direct preference optimisation.
How to build it
Format data as `{"prompt": ..., "chosen": ..., "rejected": ...}`, load into a `Dataset`, and pass to `DPOTrainer(model, ref_model, beta=0.1, train_dataset=ds)` to align without a separate reward model.