Exploring the Transformer Series (30) --- Decoding Speculation
Speculative decoding, speculative sampling, blockwise parallel decoding, token tree verification, and Hugging Face implementation details.
Tag: #transformer
Exploring the Transformer Series (29) --- DeepSeek MoE
DeepSeek MoE: load balancing, fine-grained and shared experts, DeepSeek V1/V2/V3 routing, MoD, LoRA hybrids, and efficient fine-tuning.
Exploring the Transformer Series (33) --- DeepSeek MTP
DeepSeek MTP: EAGLE, HASS, classical multi-token prediction, DeepSeek’s causal-chain design, formulas, and the vLLM implementation.
Exploring the Transformer Series (35) --- Fundamentals of Large Model Quantization
Large model quantization fundamentals: outliers, superweights, massive activations, PTQ, QAT, and common quantization strategies.
Exploring the Transformer Series (36) --- Large Model Quantization Scheme
Large model quantization schemes across 8-bit, 4-bit, and low-bit settings, including LLM.int8(), ZeroQuant, SmoothQuant, GPTQ, AWQ, LLM-QAT, QLoRA, FlatQuant, SqueezeLLM, SpQR, BitNet, and OneBit.
Exploring the Transformer Series (32) --- Lookahead Decoding
Lookahead decoding: Jacobi decoding, n-gram pool, 2D window, parallel verification, and llama.cpp implementation details.
Exploring the Transformer Series (31) --- Medusa
Medusa: multi-decoding heads, tree attention, typical acceptance, sparse tree construction, training strategies, and decoding flow.
Exploring the Transformer Series (34) --- Quantitative Fundamentals
Quantization fundamentals for Transformer LLMs: compression background, numerical representations, PTQ/QAT workflows, calibration, granularity, and acceleration.
Exploring the Transformer Series (28) --- DeepSeek MLA
DeepSeek MLA: low-rank KV compression, weight absorption, decoupled RoPE, resource tradeoffs, implementation details, and conversions from GQA and MHA.
Exploring the Transformer Series (25) --- KV Cache Optimization for Handling Long Text Sequences
KV cache optimization for long text sequences: sparsification, token reuse, prefix reuse, retrieval-based schemes, and long-context KV management.
Exploring the Transformer Series (26) --- KV Cache Optimization: PD Separation or Merging
KV cache optimization through PD separation or merging: static batching, ORCA, Sarathi, DistServe, SplitWise, MemServe, TetriInfer, and Mooncake.
Exploring the Transformer Series (24) --- KV Cache Optimization
KV Cache optimization: metrics, memory crisis, formula-based compression, stage-aware optimization, memory management, and scheduling.
Exploring the Transformer Series (23) --- Length Extrapolation
Length extrapolation in Transformers and LLMs: position encoding methods, RoPE extrapolation, PI, NTK-aware interpolation, YaRN, and Giraffe.
Exploring the Transformer Series (27) --- MQA & GQA
MQA and GQA: MHA review, shared KV heads, grouped-query attention, implementation details, memory and speed tradeoffs, conversion, and optimization variants.
Exploring the Transformer Series (22) --- LoRA
LoRA: PEFT, low-rank adaptation, rank, initialization, implementation, optimization, and continual learning.
Exploring the Transformer Series (19) --- FlashAttention V2 and its Upgrade
FlashAttention V2, Flash-Decoding, Flash-Mask, and FlashAttention-3.
Exploring the Transformer Series (18) --- FlashAttention
FlashAttention, online softmax, tiling, IO-awareness, and memory-efficient exact attention.
Exploring the Transformer Series (20) --- KV Cache
Autoregressive inference redundancy, KV cache, prefill vs decode, implementation, and resource usage.
Exploring the Transformer Series (21) --- MoE
Mixture-of-Experts (MoE): conditional computation, routing, experts, load balancing, implementation, and parallel inference.
Exploring the Transformer Series (14) --- Residual Networks and Normalization
Residual connections and normalization in Transformers: ResNet intuition, BatchNorm vs LayerNorm, Pre-Norm vs Post-Norm, implementations, and recent variants.
Exploring the Transformer Series (16) --- Resource Consumption
Transformer parameter counts, memory usage, activations, FLOPs, KV cache, and optimization directions.
Exploring the Transformer Series (17) --- RoPE
RoPE positional encoding, derivation, properties, extrapolation, and implementation.
Exploring the Transformer Series (15) --- Sampling and Output
Transformer generator heads, softmax, decoding strategies, sampling parameters, logits analysis, and weight sharing.
Exploring the Transformer Series (13) --- FFN
Feed-Forward Networks (FFN) in Transformers: structure, implementation, function, knowledge utilization, and optimization evolution.
Exploring the Transformer Series (11) --- Mask
Transformer masks: padding mask, sequence/causal mask, implementation details, data flow, and advanced sample-packing strategies.
Exploring the Transformer Series (12) --- Multi-head Self-Attention
Multi-head self-attention in Transformers: motivation, principles, implementation details, and modern head-composition improvements.
Exploring the Transformer Series (9) --- Location Encoding Classification
APE vs RPE in Transformers: differences, representative methods, and relative-position design patterns.
Exploring the Transformer Series (10) --- Self-Attention
Self-attention in Transformers: principles, implementation details, scaling/softmax analysis, and modern optimization directions.
Exploring the Transformer Series (7) --- Embedding
Transformer embedding fundamentals: from vectorization to trainable embeddings, implementation details, and modern text-embedding model evolution.
Exploring the Transformer Series (8) --- Position Encoding
Transformer positional encoding: why it is needed, design evolution, sinusoidal encoding analysis, and NoPE debates.
Exploring the Transformer Series (6) --- token
Tokenization fundamentals in Transformers: vocabulary construction, tokenizers, BPE/WordPiece/Unigram, and newer token-free directions.
Exploring the Transformer Series (5) --- Training & Reasoning
Transformer training and inference in practice: teacher forcing, masks, dropout, label smoothing, learning rate scheduling, and parallelism.
Exploring the Transformer Series (3) --- Data Processing
Transformer data processing pipeline: dataset choices, vocabulary/tokenizers, batch construction, masks, and training data loading in Harvard code.
Exploring the Transformer Series (4) --- Encoder & Decoder
Transformer encoder and decoder internals: architecture, data flow, cross-attention, and decoder-only design tradeoffs.
Exploring the Transformer Series (2) --- Overall Architecture
Transformer overall architecture: workflow, attention modules, construction from Harvard code, and theoretical perspectives.
Exploring the Transformer Series (1): Attention Mechanism
A deep dive into attention mechanism foundations: seq2seq background, CNN/RNN limitations, attention principles, and historical evolution to Transformer.