Exploring the Transformer Series (26) --- KV Cache Optimization: PD Separation or Merging
KV cache optimization through PD separation or merging: static batching, ORCA, Sarathi, DistServe, SplitWise, MemServe, TetriInfer, and Mooncake.
Tag: #prefill
Exploring the Transformer Series (24) --- KV Cache Optimization
KV Cache optimization: metrics, memory crisis, formula-based compression, stage-aware optimization, memory management, and scheduling.
Exploring the Transformer Series (20) --- KV Cache
Autoregressive inference redundancy, KV cache, prefill vs decode, implementation, and resource usage.