ML
Transformer Systems The Hands-On Way Roadmap by ML Learning Lab
Transformer Systems
Roadmap Progress 36 0 0
Map view 37 nodes
01 / Sequence Foundations 6 nodes 02 / Representation And Attention 9 nodes 03 / Transformer Block And Cost Model 4 nodes 04 / Efficient Attention And Serving 8 nodes 05 / MoE, Adaptation, And Sparse Experts 3 nodes 06 / Decoding Systems 4 nodes 07 / Quantization 3 nodes Seq2Seq / RNN / CNN Foundations S Seq2Seq / RNN / CNNFoundations 1 Attention Mechanism A1 1 Attention Mechanism 2 Overall Architecture A2 2 Overall Architecture 3 Data Processing A3 3 Data Processing 4 Encoder And Decoder A4 4 Encoder And Decoder 5 Training And Inference A5 5 Training And Inference 6 Tokenization And Vocabulary A6 6 Tokenization AndVocabulary 7 Embedding Tables A7 7 Embedding Tables 8 Position Encoding A8 8 Position Encoding 9 Position Encoding Classification A9 9 Position EncodingClassification 17 RoPE A17 17 RoPE 23 Length Extrapolation A23 23 Length Extrapolation 10 Self-Attention A10 10 Self-Attention 11 Masking A11 11 Masking 12 Multi-Head Self-Attention A12 12 Multi-HeadSelf-Attention 13 Feed-Forward Network A13 13 Feed-Forward Network 14 Residual And Normalization A14 14 Residual AndNormalization 15 Sampling And Output A15 15 Sampling And Output 16 Resource Consumption A16 16 Resource Consumption 20 KV Cache A20 20 KV Cache 24 KV Cache Optimization A24 24 KV Cache Optimization 27 MQA And GQA A27 27 MQA And GQA 18 FlashAttention A18 18 FlashAttention 19 FlashAttention V2 And Upgrades A19 19 FlashAttention V2 AndUpgrades 25 Long-Text KV Optimization A25 25 Long-Text KVOptimization 26 Prefill/Decode Scheduling A26 26 Prefill/DecodeScheduling 28 DeepSeek MLA A28 28 DeepSeek MLA 21 Mixture Of Experts A21 21 Mixture Of Experts 22 LoRA A22 22 LoRA 29 DeepSeek MoE A29 29 DeepSeek MoE 30 Speculative Decoding A30 30 Speculative Decoding 31 Medusa A31 31 Medusa 32 Lookahead Decoding A32 32 Lookahead Decoding 33 DeepSeek MTP A33 33 DeepSeek MTP 34 Quantization Fundamentals A34 34 QuantizationFundamentals 35 LLM Quantization Fundamentals A35 35 LLM QuantizationFundamentals 36 Quantization Schemes A36 36 Quantization Schemes