Let's Verify Step by Step

This paper investigates how to train large language models to reason more reliably, since even state-of-the-art models regularly produce logical mistakes in multi-step reasoning. It compares outcome supervision, which provides feedback only on a final result, against process supervision, which provides feedback on each intermediate reasoning step. Because human feedback is costly, the authors carefully evaluate both approaches and use active learning to improve the efficiency of collecting process-level labels.

The central finding is that process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset, with the process-supervised model solving 78% of problems from a representative subset of the MATH test set. Active learning further improves process supervision's efficacy. To support further research, the authors release PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train their best reward model, which mattered for advancing trustworthy LLM reasoning.