---
id: 20261020-T0-06
title: "小模型词表太贵？16MB冻结词表让嵌入表不再是参数大头"
title_en: "Frozen 16 MB Vocabulary Slashes Embedding Cost for Small Language Models"
url: https://ai.daily.yangsir.net/daily/20261020-T0-06
issue_date: 2026-10-20
publish_date: 2026-10-19T04:00:00.000Z
category: research
source_name: "arXiv cs.CL (NLP)"
source_url: https://arxiv.org/abs/2610.22428
---

# 小模型词表太贵？16MB冻结词表让嵌入表不再是参数大头

一篇新论文指出，小语言模型里词表往往是最大的单一部分：Gemma 3 270M 的 2.68 亿参数中有 1.68 亿在嵌入表上，且量化后嵌入仍保留 16 位精度，导致它占了模型文件的大部分体积。作者提出为小型乃至三值语言模型使用约 16MB 的冻结词表方案，把嵌入层从可训练参数中剥离。对开发者而言，这意味着同等内存预算下能跑更大的模型，端侧和低资源部署的成本结构会明显改变。

## English Version

**Frozen 16 MB Vocabulary Slashes Embedding Cost for Small Language Models**

A new paper argues that in small language models the vocabulary is the single largest component: in Gemma 3 270M, the embedding table holds 168M of 268M parameters, and because quantization keeps embeddings at 16-bit, it also makes up most of the model file. The authors propose a frozen ~16 MB vocabulary for small and ternary language models, removing the embedding layer from trainable parameters. For developers, that means larger models within the same memory budget and a materially different cost structure for on-device and low-resource deployments.

---

**来源**：[arXiv cs.CL (NLP)](https://arxiv.org/abs/2610.22428)

**详情页**：https://ai.daily.yangsir.net/daily/20261020-T0-06

---

*智语观潮 · Daily — https://ai.daily.yangsir.net/llms.txt*