Exploring the Transformer Series (28) --- DeepSeek MLA
DeepSeek MLA: low-rank KV compression, weight absorption, decoupled RoPE, resource tradeoffs, implementation details, and conversions from GQA and MHA.
Tag: #attention
Exploring the Transformer Series (27) --- MQA & GQA
MQA and GQA: MHA review, shared KV heads, grouped-query attention, implementation details, memory and speed tradeoffs, conversion, and optimization variants.
Exploring the Transformer Series (19) --- FlashAttention V2 and its Upgrade
FlashAttention V2, Flash-Decoding, Flash-Mask, and FlashAttention-3.
Exploring the Transformer Series (18) --- FlashAttention
FlashAttention, online softmax, tiling, IO-awareness, and memory-efficient exact attention.
Exploring the Transformer Series (17) --- RoPE
RoPE positional encoding, derivation, properties, extrapolation, and implementation.
Exploring the Transformer Series (12) --- Multi-head Self-Attention
Multi-head self-attention in Transformers: motivation, principles, implementation details, and modern head-composition improvements.
Exploring the Transformer Series (8) --- Position Encoding
Transformer positional encoding: why it is needed, design evolution, sinusoidal encoding analysis, and NoPE debates.
Exploring the Transformer Series (2) --- Overall Architecture
Transformer overall architecture: workflow, attention modules, construction from Harvard code, and theoretical perspectives.
Exploring the Transformer Series (1): Attention Mechanism
A deep dive into attention mechanism foundations: seq2seq background, CNN/RNN limitations, attention principles, and historical evolution to Transformer.