Human Preferences
Human preferences are pairwise or ranked comparisons of model outputs indicating which response a human rater finds more helpful, harmless, and honest — the raw signal used to align LLMs.
What is Human Preferences?
Human preferences are pairwise or ranked comparisons of model outputs indicating which response a human rater finds more helpful, harmless, and honest — the raw signal used to align LLMs.
Human preferences are pairwise or ranked comparisons of model outputs indicating which response a human rater finds more helpful, harmless, and honest — the raw signal used to align LLMs.
Where is it used?
OpenAI's InstructGPT, Anthropic's HH-RLHF, and Llama-2/3 RLHF all use human preference datasets; the `reward` and `preference` datasets on Hugging Face Hub power DPO/RLHF training.
How to build it
Collect pairs `(prompt, chosen, rejected)` via a labelling UI like Argilla, store as a JSONL dataset, and load with `datasets.load_dataset` to feed into `DPOTrainer` or a reward model trainer.