---
id: 20261106-T0-05
title: "AGORA：用项目反应理论给LLM评审员按可靠性加权"
title_en: "AGORA Aggregates LLM Judges with Item-Response Theory"
url: https://ai.daily.yangsir.net/daily/20261106-T0-05
issue_date: 2026-11-06
publish_date: 2026-11-05T05:00:00.000Z
category: research
source_name: "arXiv cs.AI"
source_url: https://arxiv.org/abs/2611.05556
---

# AGORA：用项目反应理论给LLM评审员按可靠性加权

用LLM当评审替代人工评估已是常见做法，聚合多个评审员则是抵消单个评审偏差的标准手段。但评审池里各评审员可靠性差异很大，用于验证的人工标注集也不一致。AGORA提出基于项目反应理论的可靠性感知聚合方法，按每个评审员的可靠程度加权汇总评分，让最终评判更接近真实质量。

## English Version

**AGORA Aggregates LLM Judges with Item-Response Theory**

LLM-as-a-judge is a common substitute for costly human evaluation, and aggregating multiple judges is the standard remedy for individual judge bias. However, judge pools mix models of very different reliability, and the human-labeled validation sets are inconsistent. AGORA proposes a reliability-aware aggregation method based on item-response theory, weighting each judge's scores by its measured reliability so the final judgment tracks true quality more closely.

---

**来源**：[arXiv cs.AI](https://arxiv.org/abs/2611.05556)

**详情页**：https://ai.daily.yangsir.net/daily/20261106-T0-05

---

*智语观潮 · Daily — https://ai.daily.yangsir.net/llms.txt*