KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

Hooper, Coleman, Kim, Sehoon, Mohammadzadeh, Hiva · arXiv · 2024