HelpSteer2

HelpSteer2: Open-source dataset for training top-performing reward models

HelpSteer2 is an open-source Helpfulness Dataset (CC-BY-4.0) that supports aligning models to become more helpful, factually correct and coherent, while being adjustable in terms of the complexity and verbosity of its responses. This dataset has been created in partnership with Scale AI.

When used to tune a Llama 3.1 70B Instruct Model, we achieve 94.1% on RewardBench, which makes it the best Reward Model as of 1 Oct 2024. This reward model is available on HuggingFace in both .nemo format at Llama-3.1-Nemotron-70B-Reward or HF-compatible format at Llama-3.1-Nemotron-70B-Reward-HF

Using this reward model for RLHF (specifically, REINFORCE), we were able to align a Llama-3.1-70B-Instruct model to reach AlpacaEval 2 LC of 57.6, Arena Hard of 85.0 and GPT-4-Turbo MT-Bench of 8.98, which are known to be predictive of LMSys Chatbot Arena Elo This Instruct model is available at Llama-3.1-Nemotron-70B-Instruct as .nemo model and Llama-3.1-Nemotron-70B-Instruct-HF as a HF Transformers model.