Group Sequence Policy Optimization

beautiful 🔥

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

• StaQ it! Growing neural networks for Policy Mirror Descent (2025)
• Bingo: Boosting Efficient Reasoning of LLMs via Dynamic and Significance-based Reinforcement Learning (2025)
• On-Policy RL with Optimal Reward Baseline (2025)
• RePO: Replay-Enhanced Policy Optimization (2025)
• DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO (2025)
• Truncated Proximal Policy Optimization (2025)
• KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning (2025)

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Any open source implementation?

To address the high variance issue in token-level importance sampling and the information loss in GSPO's sequence-level approach, I propose a Subsequence-level Clipped Importance Sampling method. For a sequence
$$a=(a_1,\dots ,a_T)a\; =\; (a\_1,\; \backslash dots,\; a\_T)$$
split into K subsequences, compute weights as:
$${\rho }_{\text{sub},k}=\text{clip}(\frac{{\pi }_{\theta }(a_{\text{sub},k}\mathrm{\mid }s)}{{\pi }_{{\theta }_{\text{old}}}(a_{\text{sub},k}\mathrm{\mid }s)},1-ϵ,1+ϵ),\rho =\prod _{k=1}^K{\rho }_{\text{sub},k}\backslash rho\_\{\backslash text\{sub\},\; k\}\; =\; \backslash text\{clip\}\backslash left(\; \backslash frac\{\backslash pi\_\{\backslash theta\}(a\_\{\backslash text\{sub\},\; k\}\; |\; s)\}\{\backslash pi\_\{\backslash theta\_\{\backslash text\{old\}\}\}(a\_\{\backslash text\{sub\},\; k\}\; |\; s)\},\; 1-\backslash epsilon,\; 1+\backslash epsilon\; \backslash right),\; \backslash quad\; \backslash rho\; =\; \backslash prod\_\{k=1\}^K\; \backslash rho\_\{\backslash text\{sub\},\; k\}$$
Add a trust region constraint:
$${\mathbb{E}}_s[D_{\text{KL}}({\pi }_{{\theta }_{\text{old}}}\mathrm{\mid }\mathrm{\mid }{\pi }_{\theta })]\le \delta \backslash mathbb\{E\}\_s\; [D\_\{\backslash text\{KL\}\}(\backslash pi\_\{\backslash theta\_\{\backslash text\{old\}\}\}\; ||\; \backslash pi\_\{\backslash theta\})]\; \backslash leq\; \backslash delta$$
This reduces variance by limiting product terms, retains local information via subsequence granularity, and ensures stability with clipping and KL constraints, outperforming GSPO in flexibility and efficiency.

Awesome work!!

Thanks for the awesome work! May I ask if you have plans to release the verl code for GSPO?

Thank you so much the prompt response! It helped a lot!