Exploring the Transformer Series (28) --- DeepSeek MLA
DeepSeek MLA: low-rank KV compression, weight absorption, decoupled RoPE, resource tradeoffs, implementation details, and conversions from GQA and MHA.
Tag: #kv-cache
Exploring the Transformer Series (25) --- KV Cache Optimization for Handling Long Text Sequences
KV cache optimization for long text sequences: sparsification, token reuse, prefix reuse, retrieval-based schemes, and long-context KV management.
Exploring the Transformer Series (26) --- KV Cache Optimization: PD Separation or Merging
KV cache optimization through PD separation or merging: static batching, ORCA, Sarathi, DistServe, SplitWise, MemServe, TetriInfer, and Mooncake.
Exploring the Transformer Series (24) --- KV Cache Optimization
KV Cache optimization: metrics, memory crisis, formula-based compression, stage-aware optimization, memory management, and scheduling.
Exploring the Transformer Series (27) --- MQA & GQA
MQA and GQA: MHA review, shared KV heads, grouped-query attention, implementation details, memory and speed tradeoffs, conversion, and optimization variants.
Exploring the Transformer Series (20) --- KV Cache
Autoregressive inference redundancy, KV cache, prefill vs decode, implementation, and resource usage.
Exploring the Transformer Series (16) --- Resource Consumption
Transformer parameter counts, memory usage, activations, FLOPs, KV cache, and optimization directions.