Exploring the Transformer Series (29) --- DeepSeek MoE
DeepSeek MoE: load balancing, fine-grained and shared experts, DeepSeek V1/V2/V3 routing, MoD, LoRA hybrids, and efficient fine-tuning.
Tag: #deepseek
Exploring the Transformer Series (33) --- DeepSeek MTP
DeepSeek MTP: EAGLE, HASS, classical multi-token prediction, DeepSeek’s causal-chain design, formulas, and the vLLM implementation.
Exploring the Transformer Series (28) --- DeepSeek MLA
DeepSeek MLA: low-rank KV compression, weight absorption, decoupled RoPE, resource tradeoffs, implementation details, and conversions from GQA and MHA.