Reward Model
A reward model is a neural network (usually a transformer with a scalar head) trained on preference data to predict a scalar score reflecting how good a response is, used to guide RLHF.
What is Reward Model?
A reward model is a neural network (usually a transformer with a scalar head) trained on preference data to predict a scalar score reflecting how good a response is, used to guide RLHF.
A reward model is a neural network (usually a transformer with a scalar head) trained on preference data to predict a scalar score reflecting how good a response is, used to guide RLHF.
Where is it used?
InstructGPT and Llama-2 train a separate RM from the base model; the RM scores samples in PPO; `trl.RewardTrainer` and `RewardConfig` train RMs on Hugging Face Hub preference datasets.
How to build it
Replace the LM head with `nn.Linear(hidden_size, 1)` on a causal model, train on `(chosen, rejected)` pairs with a margin loss via `RewardTrainer`, and evaluate by checking `score(chosen) > score(rejected)`.