---
id: 20260815-T0-04
title: "LoKiFormer：局部感知注意力+解耦知识记忆，LLM预训练效率提升"
title_en: "LoKiFormer: Locality-Aware Attention with Decoupled Memory Boosts LLM Pretraining Efficiency"
url: https://ai.daily.yangsir.net/daily/20260815-T0-04
issue_date: 2026-08-15
publish_date: 2026-08-14T04:00:00.000Z
category: research
source_name: "arXiv cs.LG (ML)"
source_url: https://arxiv.org/abs/2608.12419
---

# LoKiFormer：局部感知注意力+解耦知识记忆，LLM预训练效率提升

LoKiFormer针对LLM预训练的两大低效问题：自注意力缺乏局部性的归纳偏置，以及KV缓存随序列增长。该架构引入局部感知注意力机制来捕捉局部依赖，并将长期知识解耦为独立的记忆模块。实验显示，相比标准Transformer，LoKiFormer在同等算力下预训练损失下降更快，长序列任务上表现更优，为高效训练下一代LLM提供了新思路。

## English Version

**LoKiFormer: Locality-Aware Attention with Decoupled Memory Boosts LLM Pretraining Efficiency**

LoKiFormer tackles two pretraining inefficiencies in LLMs: self-attention's lack of locality bias and growing KV cache. It introduces locality-aware attention for local dependencies and decouples long-term knowledge into a separate memory module. Experiments show faster pretraining loss reduction and better long-sequence performance compared to standard Transformers, offering a new path for efficient LLM training.

---

**来源**：[arXiv cs.LG (ML)](https://arxiv.org/abs/2608.12419)

**详情页**：https://ai.daily.yangsir.net/daily/20260815-T0-04

---

*智语观潮 · Daily — https://ai.daily.yangsir.net/llms.txt*