---
id: 20260929-T0-03
title: "生产级Text-to-SQL流水线中LLM评判器与人工标注一致率仅0.04"
title_en: "LLM-as-Judge in Production Text-to-SQL Agrees with Humans at Kappa 0.04"
url: https://ai.daily.yangsir.net/daily/20260929-T0-03
issue_date: 2026-09-29
publish_date: 2026-09-28T04:00:00.000Z
category: research
source_name: "arXiv cs.CL (NLP)"
source_url: https://arxiv.org/abs/2609.30290
---

# 生产级Text-to-SQL流水线中LLM评判器与人工标注一致率仅0.04

arXiv新论文审计了一个生产级Text-to-SQL流水线中的LLM-as-Judge可靠性。研究者发现，部署的gpt-4o-mini评判器与双作者金标准之间的一致性仅为Cohen's kappa = 0.04，几乎等同于随机一致，说明该评判器在真实生产环境中的评估能力严重不足。论文随后提出了修复方案，旨在提升LLM评判器与人工标注的对齐程度。这一发现对依赖LLM自动评估SQL生成质量的团队具有警示意义，表明未经人工验证的自动评判指标可能严重失真。

## English Version

**LLM-as-Judge in Production Text-to-SQL Agrees with Humans at Kappa 0.04**

A new arXiv paper audits LLM-as-Judge reliability in a production text-to-SQL pipeline. Researchers found the deployed gpt-4o-mini judge agreed with two-author gold at only Cohen's kappa = 0.04, essentially random agreement, indicating severe inadequacy in real production evaluation. The paper then proposes repair methods to improve alignment between LLM judges and human annotations. This finding warns teams relying on automatic LLM evaluation for SQL generation quality that unvalidated auto-metrics can be deeply misleading.

---

**来源**：[arXiv cs.CL (NLP)](https://arxiv.org/abs/2609.30290)

**详情页**：https://ai.daily.yangsir.net/daily/20260929-T0-03

---

*智语观潮 · Daily — https://ai.daily.yangsir.net/llms.txt*