RLHF
Reinforcement Learning from Human Feedback (RLHF) aligns a language model by fine-tuning it with a reward model via PPO, maximising human-preferred behaviour while staying close to the reference policy.
What is RLHF?
Reinforcement Learning from Human Feedback (RLHF) aligns a language model by fine-tuning it with a reward model via PPO, maximising human-preferred behaviour while staying close to the reference policy.
Reinforcement Learning from Human Feedback (RLHF) aligns a language model by fine-tuning it with a reward model via PPO, maximising human-preferred behaviour while staying close to the reference policy.
Where is it used?
Used to align GPT-3/4, Llama-2-Chat, and Claude; the `trl.PPOTrainer` implements the full RLHF loop with a policy model, reference model, reward model, and value model.
How to build it
Use `trl.PPOConfig` + `PPOTrainer(model, ref_model, reward_model)` from the `trl` library; feed prompts, sample responses, score with the RM, and call `ppo_trainer.step(query, response, rewards)`.