---
id: 20261003-T0-08
title: "Format-Aware Fusion：让FP4预训练真正提速，抵消量化开销"
title_en: "Format-Aware Fusion Makes FP4 Pretraining Actually Faster"
url: https://ai.daily.yangsir.net/daily/20261003-T0-08
issue_date: 2026-10-03
publish_date: 2026-10-02T04:00:00.000Z
category: research
source_name: "arXiv cs.LG (ML)"
source_url: https://arxiv.org/abs/2610.00053
---

# Format-Aware Fusion：让FP4预训练真正提速，抵消量化开销

四比特浮点（FP4）Tensor Core 可加速矩阵乘法，但缩放计算、操作数打包、布局构建以及反向传播状态的保存会抵消收益。研究提出 format-aware fusion，把每种量化格式与计算流程协同设计，在预训练中减少这些额外开销。该方法针对 FP4 预训练的实际瓶颈，目标是让 FP4 的加速在端到端训练中体现出来，而不是只在矩阵乘法层面。对训练大模型的团队来说，这意味着更低的显存和更快的预训练速度。

## English Version

**Format-Aware Fusion Makes FP4 Pretraining Actually Faster**

Four-bit floating-point (FP4) Tensor Cores accelerate matrix multiplication, but scale computation, operand packing, layout construction, and saved backward state can erase the gains. The paper presents format-aware fusion, which co-designs each quantization format with the computation pipeline to reduce this overhead during pretraining. The method targets the real bottlenecks of FP4 pretraining so that speedups appear end-to-end rather than only at the matrix multiplication level. For teams training large models, this means lower memory usage and faster pretraining.

---

**来源**：[arXiv cs.LG (ML)](https://arxiv.org/abs/2610.00053)

**详情页**：https://ai.daily.yangsir.net/daily/20261003-T0-08

---

*智语观潮 · Daily — https://ai.daily.yangsir.net/llms.txt*