PPO
Proximal Policy Optimisation (PPO) is a policy-gradient RL algorithm that clips the ratio of new to old policy probabilities to prevent destructive updates, used as the optimiser in RLHF.
What is PPO?
Proximal Policy Optimisation (PPO) is a policy-gradient RL algorithm that clips the ratio of new to old policy probabilities to prevent destructive updates, used as the optimiser in RLHF.
Proximal Policy Optimisation (PPO) is a policy-gradient RL algorithm that clips the ratio of new to old policy probabilities to prevent destructive updates, used as the optimiser in RLHF.
Where is it used?
PPO is the RL engine behind InstructGPT and Llama-2-Chat alignment; `trl.PPOTrainer` implements clipped PPO with a learned value function and KL penalty to the reference model.
How to build it
Run `trl.PPOTrainer.step()` in a loop, passing query/response tensors and reward scores; the trainer internally computes clipped surrogate loss, value loss, and KL divergence against the ref model.