---
id: 20261103-T0-09
title: "Li²SAO：用组相对Actor-Critic自举，让单次Rollout异步RL训练提速"
title_en: "Li²SAO: Group-Relative Actor-Critic Bootstrapping for Single-Rollout Asynchronous RL"
url: https://ai.daily.yangsir.net/daily/20261103-T0-09
issue_date: 2026-11-03
publish_date: 2026-11-02T05:00:00.000Z
category: research
source_name: "arXiv cs.CL (NLP)"
source_url: https://arxiv.org/abs/2610.40726
---

# Li²SAO：用组相对Actor-Critic自举，让单次Rollout异步RL训练提速

论文提出Li²SAO方法，针对异步强化学习在LLM后训练中的效率问题：异步RL在rollout到达时即更新模型，但单次rollout样本的利用效率受限。该方法通过组相对actor-critic自举，在单rollout条件下提升异步RL优化的效果。对训练侧的意义在于，可以用更少的采样开销维持异步后训练的稳定性，降低长周期RL训练的算力成本。

## English Version

**Li²SAO: Group-Relative Actor-Critic Bootstrapping for Single-Rollout Asynchronous RL**

The paper introduces Li²SAO, targeting efficiency in asynchronous reinforcement learning for LLM post-training. Async RL updates the model as rollouts arrive, but the value of each single-rollout sample is limited. Li²SAO uses group-relative actor-critic bootstrapping to improve asynchronous RL optimization under single-rollout conditions. The practical implication: maintain stability in async post-training with fewer sampling costs, lowering compute expense for long RL training runs.

---

**来源**：[arXiv cs.CL (NLP)](https://arxiv.org/abs/2610.40726)

**详情页**：https://ai.daily.yangsir.net/daily/20261103-T0-09

---

*智语观潮 · Daily — https://ai.daily.yangsir.net/llms.txt*