--- license: apache-2.0 pipeline_tag: text-generation library_name: transformers --- # ExGRPO: Learning to Reason from Experience This repository contains the model based on the **ExGRPO** framework, presented in the paper [ExGRPO: Learning to Reason from Experience](https://huggingface.co/papers/2510.02245). ExGRPO (Experiential Group Relative Policy Optimization) is a novel paradigm for improving the reasoning ability of large language models (LLMs) by addressing the computational inefficiency and instability of standard on-policy training in Reinforcement Learning from Verifiable Rewards (RLVR). ExGRPO strategically manages and replays high-value experiences through bucketed prioritization and mixed-policy optimization, enabling more efficient and stable RLVR training. Code: [https://github.com/ElliottYan/LUFFY/tree/main/ExGRPO](https://github.com/ElliottYan/LUFFY/tree/main/ExGRPO)
ExGRPO Overview
--- ## Introduction Existing RLVR methods for reasoning tasks predominantly rely on on-policy optimization, which discards online rollouts after a single update, wasting valuable exploration signals and constraining scalability. We conduct a systematic analysis of experience utility in RLVR and identify question difficulty and trajectory entropy as effective online proxies for assessing experience quality. Building on these insights, we propose *ExGRPO*, a novel framework that **strategically manages and replays high-value experiences** through bucketed prioritization and mixed-policy optimization, enabling more efficient and stable RLVR training. ### Key Highlights: - **Experience Value Modeling**: Introduces the online proxy metrics: rollout correctness and trajectory entropy, for quantifying the value of RLVR experience. - **ExGRPO Framework**: Built on top of GRPO, ExGRPO introduces a systematic experience management mechanism and an experience optimization objective to maximize the benefit of past explorations. - **Generalization and Stability**: Demonstrates broad applicability across different backbone models and mitigates training collapse of on-policy RLVR in challenging scenarios. --- ## Getting Started ### Installation You can install dependencies by running the following commands: ```bash conda create -n exgrpo python=3.10 conda activate exgrpo cd exgrpo pip install -r requirements.txt pip install -e . cd verl pip install -e . ``` > **Note**: If you encounter issues caused by the `pyairports` library, please refer to this hot-fix [solution](https://github.com/ElliottYan/LUFFY?tab=readme-ov-file#update-98). For the `flash-attn` library, we use the `v2.7.4-post1` release and recommend installing it via the pre-built wheel. Please adjust based on your environment. ```bash wget https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.4.post1/flash_attn-2.7.4.post1+cu12torch2.4cxx11abiFALSE-cp310-cp310-linux_x86_64.whl pip install flash_attn-2.7.4.post1+cu12torch2.4cxx11abiFALSE-cp310-cp310-linux_x86_64.whl ``` ## Usage ### Data Preparation You need to first run the data preparation script to get the training data in parquet format. ```bash cd data python prepare_train.py --dataset_name Elliott/Openr1-Math-46k-8192 --output_file openr1.parquet ``` > **Note**: Although we utilize the OpenR1 data, only the question field is used in RLVR. The ExGRPO data processing pipeline does not incorporate the external R1 trajectory during training. --- ## Evaluation ## Main Results ### Zero RLVR on Qwen2.5-Math-7B & Continual RLVR on LUFFY
Main Results
### Zero RLVR on Llama3.1-8B (Base, Instruct), Qwen2.5-Math 1.5B Base, Qwen2.5-7B Instruct
Model Extensions Bar
Click to view full results of model extension
Full Model Extensions
## Released Models | **Model** | **Huggingface** | **Base Model** | |-----------------------------------|------------------|------------------| | ExGRPO-Qwen2.5-Math-7B-Zero | https://huggingface.co/rzzhan/ExGRPO-Qwen2.5-Math-7B-Zero | Qwen2.5-Math-7B | | ExGRPO-LUFFY-7B-Continual | https://huggingface.co/rzzhan/ExGRPO-LUFFY-7B-Continual | LUFFY-Qwen-Math-7B-Zero | | ExGRPO-Qwen2.5-7B-Instruct | https://huggingface.co/rzzhan/ExGRPO-Qwen2.5-7B-Instruct | Qwen2.5-7B Instruct | | ExGRPO-Qwen2.5-Math-1.5B-Zero | https://huggingface.co/rzzhan/ExGRPO-Qwen2.5-Math-1.5B-Zero | Qwen2.5-Math-1.5B | | ExGRPO-Llama3.1-8B-Zero | https://huggingface.co/rzzhan/ExGRPO-Llama3.1-8B-Zero | Llama3.1-8B | | ExGRPO-Llama3.1-8B-Instruct | https://huggingface.co/rzzhan/ExGRPO-Llama3.1-8B-Instruct | Llama3.1-8B Instruct | --- ## Citation If you find our model, data, or evaluation code useful, please kindly cite our paper: ```bib @article{zhan2025exgrpo, title={ExGRPO: Learning to Reason from Experience}, author={Runzhe Zhan and Yafu Li and Zhi Wang and Xiaoye Qu and Dongrui Liu and Jing Shao and Derek F. Wong and Yu Cheng}, year={2025}, journal = {ArXiv preprint}, volume = {2510.02245}, url={https://arxiv.org/abs/2510.02245}, } ```