---
id: 20261112-T0-07
title: "新分词器覆盖204种语言，token数比GPT-4o更少，英语也不例外"
title_en: "New Tokenizer Covers 204 Languages and Uses Fewer Tokens Than GPT-4o's, Even on English"
url: https://ai.daily.yangsir.net/daily/20261112-T0-07
issue_date: 2026-11-12
publish_date: 2026-11-11T05:00:00.000Z
category: research
source_name: "arXiv cs.CL (NLP)"
source_url: https://arxiv.org/abs/2611.13059
---

# 新分词器覆盖204种语言，token数比GPT-4o更少，英语也不例外

研究团队用约200种语言的人类书写文本从头训练分词器，推出Havon v3。实测显示，GPT-4o的分词器处理同样内容时，印度各语言所需的token数是英语的1.6到14倍，直接推高了服务成本和延迟。Havon v3覆盖204种语言，token数少于GPT-4o的分词器，英语上同样占优。这意味着多语言应用的推理成本有望下降，尤其是印度语言等此前被分词器拖累的场景。

## English Version

**New Tokenizer Covers 204 Languages and Uses Fewer Tokens Than GPT-4o's, Even on English**

Researchers trained tokenizers from scratch on human-written text in roughly 200 languages, producing Havon v3. On GPT-4o's tokenizer, Indian languages require 1.6 to 14 times as many tokens as English for the same content, driving up serving cost and latency. Havon v3 covers 204 languages and needs fewer tokens than GPT-4o's tokenizer, including on English. The result points to lower inference costs for multilingual applications, particularly for Indian-language use cases that current tokenizers handle inefficiently.

---

**来源**：[arXiv cs.CL (NLP)](https://arxiv.org/abs/2611.13059)

**详情页**：https://ai.daily.yangsir.net/daily/20261112-T0-07

---

*智语观潮 · Daily — https://ai.daily.yangsir.net/llms.txt*