---
id: 20260722-T0-01
title: "RL训练大模型推理，8个额外token就能让弱模型监督强模型"
title_en: "8 Tokens Enable Weak-to-Strong RL Supervision for LLMs"
url: https://ai.daily.yangsir.net/daily/20260722-T0-01
issue_date: 2026-07-22
publish_date: 2026-07-21T04:00:00.000Z
category: research
source_name: "arXiv cs.AI"
source_url: https://arxiv.org/abs/2607.16205
---

# RL训练大模型推理，8个额外token就能让弱模型监督强模型

arXiv 新论文提出一种弱到强的离线策略强化学习方法（Weak-to-Strong Off-Policy RL），通过在推理时添加8个辅助分支token，让能力较弱的模型也能有效监督和训练更强的模型。该方法显著降低了强化学习中对强验证模型的依赖，为语言模型推理能力的扩展提供新路径。研究人员发现，仅需少量额外计算开销，即可实现性能对齐。

## English Version

**8 Tokens Enable Weak-to-Strong RL Supervision for LLMs**

A new arXiv paper introduces Weak-to-Strong Off-Policy RL, using just 8 auxiliary branch tokens to let weaker models supervise stronger ones during RL training. The method reduces reliance on strong verifiers and enables scalable reasoning improvements with minimal computational overhead.

---

**来源**：[arXiv cs.AI](https://arxiv.org/abs/2607.16205)

**详情页**：https://ai.daily.yangsir.net/daily/20260722-T0-01

---

*智语观潮 · Daily — https://ai.daily.yangsir.net/llms.txt*