Exploring the Transformer Series (30) --- Decoding Speculation
Speculative decoding, speculative sampling, blockwise parallel decoding, token tree verification, and Hugging Face implementation details.
Tag: #inference
Exploring the Transformer Series (35) --- Fundamentals of Large Model Quantization
Large model quantization fundamentals: outliers, superweights, massive activations, PTQ, QAT, and common quantization strategies.
Exploring the Transformer Series (32) --- Lookahead Decoding
Lookahead decoding: Jacobi decoding, n-gram pool, 2D window, parallel verification, and llama.cpp implementation details.
Exploring the Transformer Series (31) --- Medusa
Medusa: multi-decoding heads, tree attention, typical acceptance, sparse tree construction, training strategies, and decoding flow.
Exploring the Transformer Series (25) --- KV Cache Optimization for Handling Long Text Sequences
KV cache optimization for long text sequences: sparsification, token reuse, prefix reuse, retrieval-based schemes, and long-context KV management.
Exploring the Transformer Series (26) --- KV Cache Optimization: PD Separation or Merging
KV cache optimization through PD separation or merging: static batching, ORCA, Sarathi, DistServe, SplitWise, MemServe, TetriInfer, and Mooncake.
Exploring the Transformer Series (24) --- KV Cache Optimization
KV Cache optimization: metrics, memory crisis, formula-based compression, stage-aware optimization, memory management, and scheduling.
Exploring the Transformer Series (20) --- KV Cache
Autoregressive inference redundancy, KV cache, prefill vs decode, implementation, and resource usage.
Exploring the Transformer Series (16) --- Resource Consumption
Transformer parameter counts, memory usage, activations, FLOPs, KV cache, and optimization directions.